Early Stopping for Large Reasoning Models via Confidence Dynamics

1. Problem Statement

Large Reasoning Models (LRMs) utilize long Chain-of-Thought (CoT) generation to solve complex problems. However, this approach incurs substantial computational costs and can lead to performance degradation due to "overthinking."

• The Core Challenge: Determining the optimal moment to terminate reasoning and output a final answer.
• Limitations of Existing Methods:
  • Training-time methods: Require additional training or model compression, limiting inference-time applicability.
  • Existing Inference-time methods: Rely on intermediate answer predictions or fixed confidence thresholds. While effective for stopping when an answer is found, they often fail to detect unproductive reasoning (e.g., loops, unstable progress) where the model continues generating tokens despite not improving, leading to unnecessarily long generations.

2. Key Observations

The authors analyzed the confidence dynamics of intermediate answers during reasoning and identified two distinct behaviors:

1. Correct Trajectories: Typically reach high confidence early in the generation process. Once the model is confident, subsequent reasoning steps are often redundant.
2. Incorrect Trajectories: Tend to be significantly longer (often double the token count of correct ones) and exhibit unstable confidence dynamics. Instead of converging, confidence fluctuates or fails to improve consistently, even as the generation length increases.
3. Temporal Signal: Early-stage reasoning steps provide the most reliable signals for distinguishing between correct and incorrect trajectories. Late-stage confidence is often unreliable because models tend to become increasingly confident even on incorrect paths.

3. Methodology: CoDE-Stop

The authors propose CoDE-Stop (Confidence Dynamics Early Stopping), a training-free inference-time method that leverages the temporal evolution of confidence to decide when to stop. It combines two complementary signals:

A. Ramping Confidence Threshold ($r_k$)

To identify when a model has reached a reliable answer, CoDE-Stop uses a dynamic threshold that increases over time:
$$r_k = \min\left(r_{max}, r_{min} + \frac{r_{max} - r_{min}}{\text{steps}} \cdot k\right)$$

• This ensures that the model is allowed to explore early on (lower threshold) but requires higher confidence to stop as the reasoning process progresses.

B. Degeneration Score ($D_k$)

To detect unproductive reasoning (instability), CoDE-Stop calculates an accumulated "degeneration score" at each step $k$:
$$D_k = \sum_{i=1}^{k} w_i v_i$$

• Instability Indicator ($v_i$): A binary signal indicating if confidence is low and failing to improve relative to the previous step. Specifically, $v_i = 1$ if $2c_i - c_{i-1} < \delta$ (where $c_i$ is the confidence at step $i$ and $\delta$ is a threshold). This captures both low confidence and sudden drops.
• Weighting Function ($w_i$): Assigns higher importance to earlier steps, based on the observation that early signals are more informative. The function is defined as $w_i = \log(\frac{T_k}{T_i}) + 1$, where $T$ represents token positions.

Stopping Condition

Generation terminates at step $k$ if either:

1. $c_k \geq r_k$ (High confidence reached), OR
2. $D_k \geq \tau$ (Accumulated instability detected).

4. Experimental Setup

• Models: Evaluated on diverse open-source reasoning models: Qwen3-4B/14B, DeepSeek-R1-Distill-Llama-8B, and Llama-3.1-Nemotron-Nano-8B-v1.
• Benchmarks: AIME 2024/2025, MATH500, GSM8K, and GPQA-Diamond.
• Baselines: Compared against state-of-the-art inference-time early stopping methods (DEER, Think or Not, EAT, RCPD, Answer Convergence) and a Vanilla full-length baseline.
• Metrics: Accuracy, Average Reasoning Length (Tokens), Compression Rate, and Total Token Compute (including overhead).

5. Key Results

• Accuracy-Compute Tradeoff: CoDE-Stop consistently achieves a superior Pareto frontier compared to all baselines. It reduces total token usage by 25–50% compared to standard full-length reasoning while maintaining accuracy comparable to strong baselines.
• Handling Unproductive Reasoning: Unlike methods like DEER (which rely on fixed thresholds), CoDE-Stop effectively terminates long, unproductive incorrect trajectories by detecting instability via the degeneration score.
• Robustness: The method performs well across different prompting strategies (Vanilla, Budget Forcing, Chain-of-Draft, NoThinking) and is robust to the choice of reasoning step delimiters (e.g., "Wait" vs. "Alternatively").
• Ablation Studies:
  • Using trajectory length alone as a stopping signal is insufficient; the confidence dynamics (degeneration score) are crucial.
  • Weighting early steps more heavily ($w_i$) significantly outperforms uniform or inverse weighting.
  • The method exhibits a smooth tradeoff curve controlled by the threshold $\tau$, making it easy to tune for specific compute budgets.

6. Significance and Contributions

• Training-Free Efficiency: CoDE-Stop offers a simple, plug-and-play solution that requires no additional model training or auxiliary predictors, making it immediately deployable.
• New Insight into Reasoning: The paper provides a fundamental analysis of confidence dynamics, revealing that early-stage instability is a stronger predictor of failure than late-stage confidence levels.
• Practical Impact: By reducing token usage by up to 50% without sacrificing accuracy, CoDE-Stop significantly lowers the inference cost and latency of Large Reasoning Models, addressing a critical bottleneck for their scalable deployment.
• Future Direction: The work suggests that simple heuristics derived from model outputs (confidence) are powerful, but richer signals may exist in model activations, pointing toward future research in extracting internal state signals for early stopping.