Technical Summary: Scaling with Confidence

Problem Statement

While Reinforcement Learning (RL) has significantly advanced Large Language Models (LLMs) in reasoning and question-answering, prevailing RL reward designs typically prioritize response correctness while neglecting confidence calibration. This oversight leads to a critical failure mode: models often become overconfident in their incorrect outputs (hallucinations) when uncertain. Existing calibration methods face three primary limitations:

1. Coarse Granularity: They often reduce calibration to a binary classification (certain vs. uncertain), failing to capture nuanced uncertainty.
2. Accuracy-Calibration Trade-off: Improvements in calibration frequently come at the expense of accuracy, as models may learn to provide deliberately incorrect low-confidence answers to minimize calibration loss.
3. Underutilization: Calibrated confidence is rarely leveraged for downstream decision-making, such as adaptive inference and resource allocation.

Methodology

The paper proposes a unified framework consisting of two core components: C3RL (training) and CAS (inference).

1. Correctness and Confidence Calibration Reinforcement Learning (C3RL)

C3RL is a novel RL algorithm designed to jointly optimize correctness and calibration without sacrificing performance. It utilizes a composite reward function ($R_{C3RL}$) comprising three distinct parts:

• Correctness Reward ($R_{correctness}$): A standard indicator reward ($+1$ for correct, $0$ for incorrect) ensuring the model updates in the direction of factual accuracy, preventing "calibrated but wrong" shortcuts.
• Calibration Reward ($R_{calibration}$): This component incentivizes the alignment between verbalized confidence and actual correctness. Using a threshold $t=5$, the model is rewarded ($+\beta$) for "certain correct" and "uncertain incorrect" responses, and penalized ($-\beta$) for "certain incorrect" and "uncertain correct" responses.
• Reference Accuracy Reward ($R_{reference}$): To prevent the model from hallucinating on questions where it originally had a high probability of being correct, this reward utilizes dataset-informed reference tags.
  • If a question was originally "all incorrect" (based on base model sampling) but the model answers correctly, it receives a positive reward ($+\alpha$).
  • If a question was originally "all correct" but the model hallucinates an incorrect answer, it receives a negative reward ($-\alpha$).

Training Data Construction: The authors constructed a high-quality dataset from mathematics, science, and logic reasoning domains. For each training sample, the base model generates 10 responses to assign a reference accuracy tag: "all correct," "partially correct," or "all incorrect."

2. Confidence-based Adaptive Test Time Scaling (CAS)

Leveraging the well-calibrated verbalized confidence produced by C3RL, the authors introduce CAS, an inference-time strategy that dynamically allocates computational resources.

• Mechanism: Instead of relying on frequency or entropy (which are often uncalibrated), CAS uses the sum of verbalized confidence scores for unique answers.
• Stopping Criterion: The algorithm calculates the probability that the most confident answer ($c_1$) dominates the second most confident answer ($c_2$) using a Beta-distribution-based formula.
  • If $Prob(c_1, c_2) \geq P_{stop}$ (indicating the top answer is highly dominant), sampling stops early.
  • Otherwise, sampling continues.
• Resource Allocation: This strategy effectively shifts computational budget from high-confidence samples (which require fewer samples to confirm) to low-confidence samples (which benefit more from additional sampling).

Key Contributions

1. C3RL Algorithm: A novel RL approach that simultaneously improves correctness and calibration, preserving overall performance while reducing overconfidence. It is the first to integrate reference accuracy information into the reward function to stabilize learning.
2. CAS Strategy: An early-stopping mechanism for test-time scaling that utilizes calibrated confidence to dynamically allocate inference budgets, dedicating more computation to uncertain samples.
3. Unified Framework: The synergy of C3RL and CAS creates a coherent pipeline enabling LLMs to express calibrated confidence and adaptively control inference costs.

Experimental Results

The framework was evaluated across 8 text and multimodal datasets (including AGIEval, MMLU, MMMU, and MathVista) using Qwen2.5VL-7B-Instruct and Llama-3.2-3B-Instruct.

• Calibration vs. Accuracy: C3RL outperforms the current state-of-the-art RL method (RLCR) and other baselines (SaySelf, RLVR) in balancing performance and reliability. On out-of-domain (OOD) datasets, C3RL achieves superior calibration (lower Expected Calibration Error) while maintaining or surpassing the accuracy of RLCR. Notably, C3RL avoids the accuracy drop seen in other calibration-focused methods.
• Inference Efficiency: In test-time scaling experiments, CAS significantly reduces the inference budget required to match the accuracy of standard Majority Voting (MV).
  • CAS reduced the number of required samples by up to 12.33 times on OOD text datasets compared to MV.
  • CAS consistently outperformed Adaptive-Consistency (AC), using less than half the sampling budget of AC on OOD text tasks while maintaining competitive accuracy.
• Stratification Analysis: Experiments confirmed that low-confidence samples benefit significantly more from increased sampling (jumping from ~49% to ~54.5% accuracy) compared to high-confidence samples (which showed negligible gains), validating the resource-shifting logic of CAS.

Significance and Claims

The authors claim that their work addresses the critical gap between model performance and reliability. By integrating correctness, calibration, and reference accuracy, C3RL enables models to "signal low confidence when they are uncertain" without sacrificing their ability to answer correctly.

The significance of the work lies in the synergy between training and inference:

• C3RL provides the necessary "well-calibrated verbalized confidence."
• CAS efficiently leverages this signal to optimize resource usage.

The paper concludes that this unified pipeline paves the way for deploying more reliable and resource-efficient LLMs, particularly in safety-critical domains where avoiding overconfident hallucinations is paramount. The authors note limitations, including potential training bias due to data scarcity in science domains and the fact that C3RL's calibration, while superior to accuracy-focused RL methods, still trails behind pure calibration methods like SaySelf in isolation.