On the Optimal Reasoning Length for RL-Trained Language Models

Technical Summary: On the Optimal Reasoning Length for RL-Trained Language Models

Problem Statement
Reinforcement learning (RL) applied during post-training has significantly enhanced the reasoning capabilities of large language models (LLMs), particularly through extended chain-of-thought (CoT) generation. However, a critical side effect is that RL-trained models tend to produce increasingly verbose outputs, escalating both training and inference costs. While recent literature has proposed various length-control methods to penalize excessive generation, the precise relationship between output length and reasoning accuracy remains unclear. Specifically, it is unknown whether the non-monotonic relationship between reasoning length and correctness—previously observed in test-time interventions on fixed models—also manifests across RL-trained policies that differ only in their length-control configurations. Furthermore, the underlying mechanism driving accuracy degradation in long-output regimes has not been fully decomposed.

Methodology
The authors conducted a controlled empirical study to investigate the length–accuracy relationship across multiple models and tasks.

• Models and Tasks: Experiments were performed on three models: DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-1.7B, and Qwen3-4B-Base. Tasks included mathematical reasoning (benchmarks: AIME 2024, AIME 2025, AMC, MATH-500) and code generation (HumanEval, MBPP+).
• Training Setup: The training followed the DAPO framework with modifications: overlong reward shaping was disabled to isolate the effects of specific length-control methods, and PPO mini-batch sizes were set equal to generation batch sizes for fully on-policy updates.
• Length-Control Methods: The study compared several length-control strategies by sweeping their primary hyperparameters to generate a family of policies with varying average output lengths. Methods included:
  • Baselines: Sample Avg (GRPO) and Token Avg (DAPO), which normalize gradients without explicit length penalties.
  • Explicit Penalties: RLOO-LP (reward shaping based on length), ALP (accuracy-based length penalty), and DRPO (length-based weighting within the DisCO framework).
• Evaluation Metrics: Beyond standard sample accuracy, the authors introduced a decomposition of the answer distribution:
  • Mode Accuracy: The fraction of problems where the most frequently sampled answer (the mode) is correct. This measures the correctness of the distribution's center.
  • Mode Leakage: The average fraction of samples falling outside the modal group. This measures the dispersion of samples around the center.

Key Results

1. Non-Monotonic Length–Accuracy Relationship: Across all models, benchmarks, and length-control methods, sample accuracy is non-monotonic with respect to average output length. Accuracy initially rises with length, reaches a peak at an intermediate value, and then plateaus or declines as length increases further. This pattern holds for both mathematical reasoning and code generation.
2. Divergence Between Mode and Sample Accuracy: A critical finding is the divergence between mode accuracy and sample accuracy in long-output regimes. While sample accuracy declines or saturates, mode accuracy continues to improve as output length increases. This indicates that the "center" of the answer distribution is becoming more correct even as individual samples become less reliable.
3. Mechanism of Degradation: The decline in sample accuracy at long lengths is driven by increased dispersion (mode leakage). In the long-output regime, samples spread further away from an increasingly correct central mode. The net effect is that gains from a more accurate center are eventually offset by the loss in concentration of the samples.
4. U-Shaped Dispersion: Mode leakage exhibits a U-shaped relationship with length in several settings. At short lengths, leakage is high due to a lack of convergence on a dominant answer. At optimal lengths, leakage is minimized. At excessive lengths, leakage rises again as samples scatter around a correct but less concentrated mode.

Key Contributions

• Generalization of Non-Monotonicity: The paper demonstrates that the non-monotonic relationship between output length and accuracy, previously documented for test-time interventions on fixed models, also arises across a controlled comparison of RL-trained policies spanning multiple models, benchmarks, and length-control methods.
• Decomposition of Answer Distribution: The authors decompose the answer distribution into the correctness of its center (mode accuracy) and the dispersion of samples (mode leakage). They show that longer outputs improve the former while increasing the latter, providing a mechanistic explanation for the puzzling divergence between mode accuracy and sample accuracy.

Significance and Claims
The paper claims to resolve the ambiguity surrounding the efficacy of length control in RL-trained reasoning models. It establishes that simply extending reasoning does not uniformly improve performance; there exists an optimal intermediate length. The study clarifies that the degradation in performance at excessive lengths is not due to the model learning incorrect reasoning paths (the center remains correct or improves), but rather due to the model becoming less confident or consistent in its sampling (increased dispersion).

The authors conclude that the optimal penalty strength for length control varies across methods and models, suggesting a need for future work to develop automated methods for identifying the optimal length regime without manual hyperparameter search. The work does not propose new length-control algorithms but rather provides a diagnostic framework for understanding the trade-offs inherent in current methods.