Home / AI / Reevaluating RL Progress: Study Reveals Gains Often Mirrored by Random Variance

Reevaluating RL Progress: Study Reveals Gains Often Mirrored by Random Variance

abril 20, 2025 10:16 pm

Reasoning abilities have become a focal point in the evolution of large language models (LLMs), playing a pivotal role in leading AI systems from prominent research institutions. Although there is a growing body of research aimed at comprehending and improving the reasoning capabilities of LLMs, there are still substantial challenges in accurately evaluating these capabilities. Concerns about evaluation rigor are rising as non-reproducible or inconclusive assessments can distort scientific understanding, misguide adoption choices, and bias future research directions. In the fast-evolving realm of LLM reasoning, where rapid publication cycles and benchmarking contests are common, methodological shortcuts can inadvertently hinder genuine progress. Documented reproducibility issues in LLM evaluations, particularly in reasoning tasks, necessitate increased attention and stricter evaluation standards to ensure that reported advancements truly reflect capabilities and not flaws in assessment methodologies.

Numerous strategies have been developed to enhance reasoning capabilities in language models, with supervised fine-tuning (SFT) and reinforcement learning (RL) being key methods. Recent innovations have built upon the DeepSeek-R1 approach through novel RL algorithms like LCPO, REINFORCE++, DAPO, and VinePPO. Researchers have also conducted empirical studies examining RL design spaces, data scaling trends, curriculums, and reward mechanisms. Despite these advancements, the field continues to face significant evaluation challenges. Progress in machine learning often lacks thorough assessment, with many reported improvements failing to withstand testing against well-tuned baselines. RL algorithms are especially vulnerable to variations in implementation specifics, including random seeds, leading to questions about the reliability of benchmarking practices.

Prompted by inconsistent claims in reasoning research, a study by researchers from Tübingen AI Center, University of Tübingen, and University of Cambridge undertakes a rigorous examination of mathematical reasoning benchmarks, revealing that many recent empirical conclusions fall apart upon careful re-evaluation. The analysis highlights surprising sensitivity in LLM reasoning models to minor design choices, such as decoding parameters, prompt formatting, random seeds, and hardware setups. Small benchmark sizes significantly contribute to this instability, with individual questions potentially altering Pass@1 scores by over 3 percentage points on datasets like AIME’24 and AMC’23, leading to performance variations across seeds that undermine published results. The study systematically investigates these stability issues and suggests best practices for enhancing reproducibility and rigor in reasoning evaluations, offering a standardized framework for re-evaluating recent techniques under more controlled conditions.

The study investigates design factors affecting reasoning performance in language models using a standardized experimental framework. Nine widely-used models across 1.5B and 7B parameter classes were evaluated, including DeepSeek-R1-Distill variants, DeepScaleR-1.5B, II-1.5 B-Preview, OpenRS models, S1.1-7B, and OpenThinker7B. Consistent hardware (A100 GPU, AMD CPU) and software configurations were used to benchmark models on AIME’24, AMC’23, and MATH500 datasets with Pass@1 metrics. The analysis found significant performance variance across random seeds, with standard deviations ranging from 5 to 15 percentage points, highlighting instability, especially in smaller datasets where a single question can shift performance by 2.5-3.3 percentage points, rendering single-seed evaluations unreliable.