🤖 AI

Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards

This paper proposes ACOER, a novel reward mechanism that stabilizes efficiency training in large reasoning models by isolating length penalties to correct answers and employing dynamic budget normalization, thereby preventing reward collapse while significantly improving accuracy and reducing token generation.

Original authors: Jungseob Lee, Seungyoon Lee, Seongtae Hong, Minhyuk Kim, Chanjun Park, Heuiseok Lim

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jungseob Lee, Seungyoon Lee, Seongtae Hong, Minhyuk Kim, Chanjun Park, Heuiseok Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Thinker" Student

Imagine you have a brilliant student (the AI model) who is great at solving math problems. However, this student has a bad habit: they talk way too much. Even for a simple problem like "2 + 2," they might write a 5,000-word essay explaining every possible way to add numbers before finally saying "4."

This is called verbosity. It wastes time and computer power. Researchers want to teach this student to be concise—to say "4" quickly without the 5,000-word essay.

The Failed Attempt: The "Strict Teacher"

To fix this, researchers tried a method called GRPO (Group Relative Policy Optimization). They added a rule: "If you write too many words, you get a penalty."

However, they made a critical mistake in how they applied this rule. They penalized both right answers that were too long and wrong answers that were too long.

The Analogy: Imagine a teacher telling a student:

"If you get the answer wrong, I will deduct points for how long your explanation was. If you get it right, I will also deduct points if you wrote too much."

What happened? The student panicked. They realized that if they wrote a long, complicated explanation and got it wrong, they got punished twice. So, they started writing nothing. They stopped thinking entirely. They would just guess "4" instantly, even if the answer was "5."

This is called Reward Collapse. The model became so afraid of the penalty for being "long and wrong" that it stopped reasoning altogether, becoming short but useless.

The Discovery: Why the "Strict Teacher" Failed

The authors of this paper investigated why this happened. They found two main reasons:

The "Wrong Answer" Trap: When you penalize long wrong answers, the math inside the AI's brain gets messed up. It creates a feedback loop where the AI thinks, "The safest bet is to say nothing at all." Even a tiny penalty for long wrong answers was enough to break the system.
The "Over-Compression" Trap: Even if you only penalize long right answers (a "Correct-Only" approach), the AI can still collapse. Sometimes, the AI gets too confident that "short is good" and starts compressing its thinking so much that it misses the solution, even for hard problems. It's like a student who memorizes the answer key but doesn't actually learn the math.

The Solution: ACOER (The "Smart Coach")

The authors propose a new method called ACOER (Adaptive Correct-Only Efficiency Reward). Think of ACOER as a smart coach who uses three specific rules to train the student without breaking them:

1. The "No Penalty for Wrong Guesses" Rule

How it works: The coach says, "If you get the answer wrong, I don't care how long your explanation was. You get zero points, but no extra penalty for being long."
Why it helps: This stops the panic. The student knows they can think deeply and make mistakes without being punished for the length of their thought process. They only get rewarded for being concise when they are right.

2. The "Moving Goalpost" Rule (Adaptive Budget)

How it works: Instead of saying "You must write under 500 words," the coach watches the student. If the student starts writing 200 words, the coach lowers the goal to 150 words. If they drop to 100, the goal becomes 80.
Why it helps: This prevents the student from getting stuck in a loop where they think "shorter is always better." It keeps the target moving so the student keeps improving gradually without suddenly giving up.

3. The "Safety Brake" Rule (Control-Loop)

How it works: The coach constantly checks the test scores. If the student starts getting answers wrong because they are writing too little, the coach immediately says, "Stop! Slow down. You can write more words again."
Why it helps: This prevents the "Over-Compression" trap. It ensures the AI never sacrifices accuracy just to be fast. If accuracy drops, the pressure to be short is relaxed.

The Results: Fast and Accurate

When they tested this new "Smart Coach" (ACOER) on difficult math problems:

Speed: The AI reduced the number of words it wrote by 62% (from thousands of words down to a manageable amount).
Accuracy: The AI kept its math skills just as good as before. It didn't start guessing randomly.
Adaptability: The AI learned to be short on easy problems (like "2+2") but would still write a long, detailed explanation for very hard problems if it needed to.

Summary

The paper teaches us that to make AI efficient, you cannot just punish it for being long, especially when it's wrong. That makes it stop thinking. Instead, you should reward it for being short only when it is right, and have a safety system that stops it from getting too short if it starts making mistakes. This creates a stable, efficient, and smart AI.

Technical Summary: Beyond Penalizing Mistakes (ACOER)

1. Problem Statement

Large Reasoning Models (LRMs) such as DeepSeek-R1 and Qwen3 achieve high performance on mathematical benchmarks by generating extended reasoning traces via Reinforcement Learning (RL). However, this capability incurs a severe efficiency bottleneck, often generating thousands of tokens even for solvable problems. To address this, recent work integrates length-penalizing rewards into Group Relative Policy Optimization (GRPO).

A critical, yet poorly understood, failure mode in this domain is reward collapse. When models are trained with length penalties, they frequently degenerate into producing outputs that are too short to reason effectively, causing a simultaneous drop in token generation and problem-solving accuracy. While practitioners recognize this instability, the specific mechanisms driving it—particularly how GRPO's group normalization interacts with length penalties on incorrect answers—remain unformalized. Furthermore, existing "correct-only" reward strategies (penalizing length only on correct answers) have shown empirical stability but lack a theoretical explanation for why they sometimes still fail stochastically.

2. Methodology and Analysis

2.1 Diagnostic Framework

The authors constructed a unified evaluation framework to isolate the triggers of reward collapse. They defined a generalized reward function decoupling length penalties for correct ( $\alpha$ ) and incorrect ( $\beta$ ) answers:
$r = c + c \cdot \alpha \cdot f(\ell) - (1-c) \cdot \beta \cdot f(\ell) + r_{format}$
where $c$ is correctness, $\ell$ is token length, and $f(\ell)$ is a length-dependent function. They evaluated 11 configurations across varying $\beta$ values (from 0 to 0.10) and signal types (continuous vs. discrete).

2.2 Mechanistic Analysis of Collapse

Through systematic evaluation and theoretical analysis, the paper identifies two distinct collapse pathways driven by GRPO's group normalization:

Pathway 1: The $\beta$ -Loop (Structural Collapse).
When continuous length penalties are applied to incorrect answers ( $\beta > 0$ ), the optimization process is structurally prone to collapse.
- Mixed Groups: In batches containing both correct and incorrect answers, incorrect samples receive a constant negative gradient with respect to length ( $\partial \hat{A}_i / \partial \ell_i < 0$ ). This creates a relentless downward pressure to truncate reasoning.
- Homogeneous Groups: In batches where all answers are incorrect, the reward variance approaches zero. Any infinitesimal difference in length causes the normalized advantage to diverge, creating explosive gradient spikes that force the policy toward zero length.
- Finding: Even a microscopic penalty weight ( $\beta=0.01$ ) is sufficient to trigger collapse.
Pathway 2: Stochastic Correct-Answer Compression.
Removing penalties for incorrect answers ( $\beta=0$ ) eliminates Pathway 1 but does not guarantee stability. If the model's correct-answer subgroup over-compresses (due to initialization or early dynamics), the same divergence logic applies within the correct subgroup, leading to stochastic collapse. Static correct-only rewards cannot reliably prevent this.

2.3 Proposed Solution: ACOER

To address both pathways, the authors propose ACOER (Adaptive Correct-Only Efficiency Reward), which employs three synergistic mechanisms:

Correct-Only Signal ( $\beta=0$ ): Eliminates the structural $\beta$ -loop by applying length penalties exclusively to correct completions.
Adaptive Budget Normalization ( $B_t$ ): Replaces the fixed maximum token limit $L$ with a dynamic budget $B_t$ based on the exponential moving average (EMA) of correct-answer lengths. This prevents runaway feedback where minor token reductions yield disproportionately large bonuses.
Control-Loop Penalty Adaptation ( $\alpha_t$ ): Dynamically adjusts the brevity bonus weight $\alpha_t$ based on accuracy preservation. If accuracy drops beyond a threshold, $\alpha_t$ decays rapidly to allow recovery, while increasing slowly when accuracy is stable.

3. Key Results

Experiments were conducted on Qwen3-1.7B trained on NuminaMath-TIR and evaluated on MATH-500, MATH-Hard, AIME 2025, and OlympiadBench.

Stability: Among methods with continuous $\beta > 0$ , the collapse rate was 100% (8/8). In contrast, ACOER and other correct-only/binary methods remained stable, with only 1/6 collapsing (due to static initialization issues).
Efficiency vs. Accuracy: On MATH-500, ACOER reduced token generation by 62% (from 5,553 to 2,134 tokens) while maintaining accuracy comparable to the base model (88.4% vs. 88.8%).
Difficulty Adaptivity: ACOER demonstrated task-adaptive behavior. It reduced tokens by 78.6% on easy problems (Level 1) but maintained a larger reasoning budget (52.6% reduction) for very hard problems (Level 5), preserving computational depth where necessary.
Comparison: ACOER outperformed collapsed methods (e.g., GRPO+LP) which suffered severe accuracy degradation (e.g., dropping to 68% accuracy) and showed superior stability compared to static correct-only baselines.

4. Significance and Claims

The paper claims to establish both the empirical and theoretical foundations of reward collapse in GRPO-based efficiency training. Its primary contributions are:

Mechanistic Explanation: It provides a formal theoretical analysis demonstrating that GRPO's group normalization inherently creates divergent advantages when continuous length penalties are applied to incorrect rollouts, explaining the inevitability of the $\beta$ -loop.
Identification of Dual Failure Modes: It distinguishes between structural collapse (driven by incorrect-answer penalties) and stochastic collapse (driven by over-compression of correct answers), showing that simply removing incorrect penalties is necessary but insufficient for absolute stability.
Stable Optimization Paradigm: ACOER offers a principled, adaptive solution that robustly prevents both failure modes, enabling efficiency-aware optimization without sacrificing reasoning capabilities.

The authors maintain a modest stance, noting that while ACOER significantly improves efficiency, its marginal accuracy gains over stable baselines (like accuracy-only GRPO) are not statistically significant given current evaluation set sizes. The primary advantage is the substantial token reduction. They also acknowledge limitations regarding model scale generalization (tested only on 1.7B), the computational cost of multi-seed analysis, and the need for further hyperparameter sensitivity exploration.

5. Limitations and Ethics

Scale: Experiments were limited to Qwen3-1.7B; generalization to larger models (7B+) requires validation.
Compute: Stochastic vulnerability analysis was limited by the high compute cost of multi-seed runs.
Ethics: The authors warn that shortening reasoning traces could reduce transparency or mask brittle reasoning. They recommend task-specific validation and monitoring, advising against safety-critical deployment based solely on token savings.