Technical Summary: Inference Time Optimization with Confidence Dynamics

Problem Statement

While inference-time optimization techniques like Best-of-N sampling and repeated sampling have significantly enhanced the reasoning capabilities of Large Language Models (LLMs), existing strategies often overlook the critical role of model uncertainty dynamics. Current state-of-the-art approaches, such as Majority Voting or confidence-weighted voting (e.g., DeepConf), rely on static confidence measures that aggregate information across an entire reasoning trace (e.g., average perplexity or sequence-level probability). These methods treat all traces equally or weight them based on a single aggregate score, potentially obscuring valuable signals regarding how a model's certainty evolves during the reasoning process. The paper posits that the trajectory of confidence from the beginning to the end of a reasoning chain contains discriminative information that is currently underutilized for answer selection.

Methodology: Confidence Dynamic Gain (CDG)

The paper introduces Confidence Dynamic Gain (CDG), a novel voting mechanism that explicitly incorporates the evolution of confidence along a reasoning trajectory.

1. Empirical Observation

Through extensive experiments on four diverse open-source architectures (DeepSeek-R1, gpt-oss, Gemma-3, Qwen-QwQ) across mathematical benchmarks (AIME, HMMT, BRUMO), the authors identified a unique pattern:

• Correct Traces: Exhibit a systematic positive confidence gain, where the model's confidence increases as the reasoning chain progresses from the initial tokens to the final conclusion.
• Incorrect Traces: Exhibit attenuated or negative confidence gains, showing declining or stagnant confidence as the reasoning proceeds.

2. Confidence Metric and Trajectory Binning

To quantify this, the authors define token-level confidence ($C_t$) using a truncated Kullback-Leibler (KL) divergence approximation based on top-$K$ token probabilities. To compare traces of varying lengths, they employ a position-normalized binning approach:

• Each trace is partitioned into $N$ equal-sized bins (default $N=10$).
• The "Head" bin represents the first $P\%$ of tokens, and the "Tail" bin represents the last $P\%$.
• Confidence Dynamic Gain ($\Delta C_\ell$) is calculated as the difference between the average confidence of the Tail bin and the Head bin:
  $$\Delta C_\ell = \bar{C}_{tail} - \bar{C}_{head}$$
  A positive $\Delta C_\ell$ indicates increasing confidence; a negative value indicates decreasing confidence.

3. The CDG Voting Algorithm

The final answer selection score ($s_\ell$) for a trace $\ell$ combines the average trace confidence ($\bar{C}_\ell$) with the dynamic gain:
$$s_\ell = \bar{C}_\ell + \beta \cdot \Delta C_\ell$$
Here, $\beta$ is a hyperparameter controlling the influence of the dynamic signal.

The final score for a candidate answer $a$ aggregates these trace scores, incorporating a count dampening factor to prevent majority voting from overwhelming the confidence signal:
$$R(a) = |T_a|^\alpha \cdot \mu_a(s_\ell)$$
Where $|T_a|$ is the count of traces for answer $a$, $\alpha \in [0, 1]$ dampens the count dominance, and $\mu_a$ is the mean score of traces producing answer $a$.

4. Theoretical Justification

The paper provides a theoretical analysis linking this phenomenon to Group Relative Policy Optimization (GRPO), a common training algorithm for reasoning models.

• Assumption: Correct traces converge to a single ground truth (high frequency at the tail), while incorrect traces diverge into diverse failure modes (low frequency at the tail).
• Mechanism: GRPO assigns positive advantages to correct traces and negative advantages to incorrect ones. This leads to a higher "tail-to-head" concentration ratio for correct traces, driving a sharp rise in confidence (logits) toward the end of the sequence. Incorrect traces, lacking this convergence, do not exhibit the same confidence spike.

Key Contributions

1. Empirical Discovery: Identification of a systematic pattern where correct reasoning traces exhibit higher confidence gains (head-to-tail) compared to incorrect traces across diverse LLM architectures.
2. Novel Method (CDG): Proposal of a training-free voting method that leverages confidence dynamics as a discriminative signal, generalizing both Majority Voting and static confidence-based methods.
3. Theoretical Insight: A theoretical analysis connecting the observed confidence gradients to the training dynamics of GRPO and verifiable rewards, explaining why correct traces naturally develop a "confidence spike" at the tail.
4. Empirical Validation: Demonstration of consistent performance improvements over state-of-the-art baselines on competitive mathematical reasoning benchmarks.

Experimental Results

The authors evaluated CDG on four models (DeepSeek-R1-8B, gpt-oss-20B, Gemma-3-27B, Qwen-QwQ-32B) across AIME 2024/2025, HMMT 2025, and BRUMO 2025.

• Performance: CDG achieved the highest average accuracy across all models and datasets.
  • It improved over Majority Voting by 5.4% on average.
  • It improved over DeepConf-Mean by 4.8%.
  • It outperformed DeepConf-Tail (which uses only tail confidence) by 1.7%, demonstrating that the dynamic aspect (head-to-tail difference) is crucial, not just the tail confidence itself.
• Ablation Studies:
  • Count Dampening ($\alpha$): Removing count dampening ($\alpha=1$) significantly degraded performance, confirming the need to balance trace counts with confidence signals.
  • Dynamic Weight ($\beta$): Performance peaked when $\beta$ was tuned to a specific range relative to the model's confidence separation, confirming that the dynamic signal provides complementary information to static confidence.
  • Trace Length: CDG maintained advantages regardless of trace length, indicating the signal is robust to sequence length variations.
  • Formatting Artifacts: Excluding \boxed{} tokens (often high-confidence formatting tokens at the end) did not alter results, proving the signal arises from reasoning dynamics rather than formatting artifacts.

Significance and Claims

The paper claims that CDG provides a robust, training-free discriminative signal for improving answer selection in LLM reasoning. By exploiting the natural dynamics of model confidence during inference, CDG offers a principled way to select correct answers over hallucinated ones without requiring model retraining or additional reward models.

The authors note that while the method is highly effective for Best-of-N scaling in mathematical reasoning, it inherits the computational costs of generating multiple traces. They also acknowledge limitations regarding the need for model-specific tuning of the $\beta$ parameter and the requirement for access to token-level log-probabilities, which may not be available in all closed-source APIs. The work suggests that these insights into confidence dynamics may generalize to other advantage-weighted policy gradient methods with verifiable rewards.