Technical Summary: SAGE – Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment

Problem Statement

Large Language Models (LLMs) frequently generate fluent, plausible text even when the underlying answer is uncertain or unsupported, leading to hallucinations with potentially severe real-world consequences. While researchers have explored internal signals (e.g., token probabilities) and sampling-based signals to estimate uncertainty, verbal uncertainty—where the model explicitly states its confidence (e.g., "I am 80% confident")—offers a direct, interpretable signal for users.

However, a critical gap exists in verbal uncertainty alignment: the expressed uncertainty often fails to reflect the model's actual behavioral reliability. Existing approaches treat uncertainty calibration as a response-level problem (e.g., supervised fine-tuning on a single output or pairwise preference optimization). The authors argue this is fundamentally insufficient because a single response cannot reveal whether the model's probability mass is concentrated on that answer or dispersed across plausible alternatives. This creates a "distributional blindness" where the model learns to mimic uncertainty expressions without learning to align them with the stability of its own response distribution.

Furthermore, while group-level targets (using repeated rollouts to estimate uncertainty) are theoretically sound, existing methods suffer from a "reward-signal bottleneck":

1. Maximum Answer Frequency (MAF): Faithful to answer stability but coarse; it ignores semantic distance and reasoning variations, mapping distinct groups to identical scores if majority counts match.
2. Semantic Entropy (SE): Handles paraphrases via hard semantic clustering but produces discontinuous, coarse rewards. Small changes within a cluster yield no signal, while crossing boundaries causes discrete jumps.
3. Kernel Language Entropy (KLE): Provides smooth signals via continuous embedding similarity but fails to preserve task-critical answer distinctions. It may treat mutually exclusive options (e.g., "A" vs. "B") or distinct numerical values as highly similar due to generic embedding geometry, leading to underestimation of uncertainty.

Methodology

The paper proposes a two-part framework to address these limitations: SAGE (Semantic-Answer Guided Entropy) for target construction and GUPO (Group-Uncertainty Preference Optimization) for training.

1. SAGE: Semantic-Answer Guided Entropy

SAGE is a group-level self-uncertainty target designed to construct an answer-conditioned uncertainty geometry. It aims to be simultaneously answer-faithful, reward-smooth, and scale-preserving.

• Linguistic Kernel: It begins with a continuous semantic similarity kernel, $K_{ling}$, based on the answer-bearing content $z_i$ of responses, measuring smooth semantic similarity.
• Answer-Equivalence Kernel: To prevent generic embeddings from blurring task-critical distinctions, SAGE introduces a task-specific binary answer-equivalence kernel, $K_{ans}^{bin}$. This kernel assigns a value of 1 to equivalent answers (e.g., same multiple-choice label, normalized numeric value, or verified symbolic expression) and 0 otherwise.
• Soft Floor: To avoid the discontinuity of hard clustering, SAGE employs a "soft floor" mechanism:
  $$K_{ans}^{soft}(a_i, a_j) = c + (1-c)K_{ans}^{bin}(a_i, a_j)$$
  where $c \in [0, 1]$ controls the degree of answer separation. $c=0$ enforces hard separation, while $c=1$ recovers pure linguistic similarity.
• Combined Kernel: The final kernel combines linguistic and answer-aware similarities:
  $$K_A(y_i, y_j) = (K_{ling}(y_i, y_j))^\alpha \cdot (K_{ans}^{soft}(a_i, a_j))^\beta$$
• Entropy Calculation: The kernel matrix is normalized into a density matrix $P_A$, and SAGE is defined as the normalized von Neumann entropy of $P_A$. Lower entropy indicates concentrated responses (low uncertainty), while higher entropy indicates dispersion across meanings, reasoning paths, or final answers.

2. GUPO: Group-Uncertainty Preference Optimization

GUPO is a training framework that utilizes SAGE as a supervision signal. Unlike standard methods that optimize the entire response, GUPO optimizes only the uncertainty channel ($v_i$) of the response $y_i = (z_i, v_i)$.

• Preference Construction: For a group of $K$ sampled responses, the reward $r_i$ is calculated based on the distance between the expressed uncertainty $\hat{u}(v_i)$ and the SAGE target $t(G_x)$.
• Optimization: GUPO applies a preference distribution over the uncertainty expressions within the group, optimizing the policy to maximize the log-likelihood of uncertainty expressions that better match the group-level target. This is formulated as:
  $$L_{GUPO} = -\mathbb{E}_{x, G_x} \left[ \sum_{i=1}^K \text{stopgrad}(p_i^{cal}) \cdot \log \pi_\theta(v_i | x, z_i) \right]$$
  where $p_i^{cal}$ is the preference probability derived from the reward scores.

Key Contributions

1. Problem Formulation: The authors identify verbal uncertainty alignment as a distributional calibration problem and pinpoint the "reward-signal bottleneck" in existing group-level targets (MAF, SE, KLE) that limits their effectiveness.
2. SAGE Target: They introduce SAGE, a novel entropy target that injects answer-aware structure into the response geometry. It preserves smoothness while correcting the failure of generic metrics to distinguish incompatible answers (e.g., different option labels or distinct numbers).
3. GUPO Framework: They propose GUPO, an uncertainty-channel preference optimization framework that decouples the calibration of verbal uncertainty from the optimization of the full response content.
4. Empirical Validation: The approach is validated across three distinct answer formats: factual QA (TriviaQA), mathematical reasoning (MATH-500), and multiple-choice reasoning (MMLU-Pro).

Results

Experiments demonstrate that GUPO + SAGE consistently outperforms existing baselines (including Direct Verbalized, CoT, CSFT, and LoVeC-DPO) and alternative group targets (MAF, SE, KLE).

• Calibration: The method achieves the lowest Brier scores and Expected Calibration Error (ECE) across all benchmarks. For instance, on MMLU-Pro, GUPO+SAGE reduced ECE to 0.113 compared to 0.179 for the next best baseline.
• Uncertainty Ranking: It significantly improves Spearman correlation, indicating that the model's expressed uncertainty better ranks examples by their actual reliability.
• Selective Accuracy: The framework yields substantial gains in high-confidence subset accuracy (e.g., +22.4 to +33.0 percentage points on MMLU-Pro at various confidence thresholds), proving that predictions assigned lower uncertainty are more likely to be correct.
• Transferability: The improvements hold across different model scales (Qwen2.5-1.5B, 3B, and 7B), suggesting the method is not dependent on a specific backbone capacity.

Significance and Claims

The paper claims that group-level behavioral uncertainty can be distilled into single-response uncertainty expressions only when both the target construction and the optimization channel are properly designed.

• Addressing the Target Gap: The authors argue that without a reliable, answer-aware target like SAGE, reward signals collapse into noise or shortcuts, preventing the optimizer from distinguishing calibrated uncertainty from superficial patterns.
• Behavioral Alignment: The work shifts the paradigm from learning to "say" uncertainty to learning to "reflect" the stability of the model's own sampling behavior.
• Practical Utility: By improving the ordering and scale of verbal uncertainty, the framework enables more effective downstream decision-making, such as trusting, verifying, or abstaining from model outputs.

Limitations: The authors modestly note that their method aligns verbal uncertainty with behavioral self-uncertainty (stability of sampling), not necessarily external correctness (truthfulness). A model can be stable but wrong. Therefore, SAGE is presented as complementary to correctness-based calibration and external validation, rather than a replacement, particularly in high-stakes domains like medicine or law where domain-specific evidence is required. Additionally, the quality of the target depends on the reliability of the answer extraction and equivalence modules.