Closing the Confidence-Faithfulness Gap in Large Language Models

1. Problem Statement

Large Language Models (LLMs) suffer from systematic verbalized miscalibration. While they often possess internal knowledge of their own accuracy, they fail to express this faithfully in natural language.

• The Phenomenon: Models tend to cluster verbalized confidence scores near the top of the range (e.g., claiming 90%+ confidence) regardless of actual performance.
• The Consequence: This is particularly dangerous for safe deployment (e.g., medical diagnosis), where a user relies on the model's stated uncertainty.
• The Gap: Existing literature suggests the information required for faithful confidence exists within the model's internal states (residual stream), but the generation process fails to "read out" or utilize this signal. Current solutions (prompt engineering, fine-tuning) treat the model as a black box rather than addressing the representational disconnect.

2. Methodology

The authors employ mechanistic interpretability tools to analyze and manipulate the internal representations of LLMs. The study focuses on three open-weight model families (Llama-3.1-8B, Qwen2.5-7B, Mistral-7B) across four datasets (MATH, MMLU, TriviaQA, TruthfulQA).

Core Techniques:

1. Gold Calibration Linear Probing:
   • The authors train ridge regression probes on residual stream activations (extracted after the MLP sublayer) to predict empirical accuracy (the fraction of times the model answers correctly across multiple samples).
   • This tests whether the model's internal "truth" signal is linearly accessible.
2. Contrastive Activation Addition (CAA) Steering:
   • To establish causality, the authors construct steering vectors by computing the difference between mean activations of high-confidence and low-confidence instances ($\delta = \mu_{high} - \mu_{low}$).
   • These vectors are injected into the residual stream during generation ($\tilde{h}_t = h_t + \alpha \hat{v}$) to causally shift the model's verbalized confidence up or down.

Experimental Design:

The study utilizes three distinct prompt types to disentangle representations:

• Pure Correctness: Model answers only (no confidence mentioned).
• Pure Confidence: Model rates confidence only (no answer provided).
• Joint: Model solves the problem and rates confidence simultaneously.

3. Key Contributions & Findings

A. Geometric Dissociation (Orthogonality)

• Finding: The model encodes internal accuracy (gold calibration) and verbalized confidence in two separate, nearly orthogonal linear directions within the residual stream.
• Evidence: The cosine similarity between the probe weights for accuracy and verbalized confidence is consistently $< 0.04$.
• Implication: The model "knows" when it is likely wrong (encoded linearly), but the generation process operates in a different subspace that fails to access this signal, leading to overconfidence.

B. The "Reasoning Contamination Effect"

• Finding: When a model is prompted to reason and rate confidence simultaneously (Joint Prompt), the relationship between the accuracy and confidence directions inverts.
• Evidence: Under pure prompts, the directions are weakly aligned (cosine $\approx +0.26$). Under joint prompts, they become sharply opposed (cosine $\approx -0.63$).
• Mechanism: The act of reasoning interferes with the confidence signal. High-magnitude activations associated with the effort of solving a difficult problem are misinterpreted by the generation process as engagement or certainty, causing the model to be most overconfident when it is actually struggling.

C. Causal Steering and Generalization

• Causality: Applying CAA steering vectors successfully shifts verbalized confidence in a controlled, approximately linear manner across different datasets and models.
• Generalization: Steering vectors derived from the MATH dataset generalize effectively to MMLU, TriviaQA, and TruthfulQA, indicating the confidence direction is a domain-general feature.
• Transfer: Vectors extracted from base models successfully steer the confidence of their instruction-tuned counterparts, suggesting that overconfidence in tuned models is a "readout failure" rather than a loss of the underlying signal.

D. Adaptive Two-Stage Steering Pipeline

To close the calibration gap, the authors propose a pipeline that avoids the "Reasoning Contamination Effect":

1. Stage 1 (Target Estimation): Use a linear probe on the pure correctness prompt (before generation) to estimate the model's internal accuracy probability. This serves as the "target" confidence.
2. Stage 2 (Adaptive Steering): Apply CAA steering during a pure confidence generation pass. The steering strength ($\alpha$) is dynamically adjusted per question to match the verbalized output to the Stage 1 target.

• Result: This approach decouples problem-solving from confidence assessment, preventing the contamination effect.

4. Results

The adaptive steering pipeline significantly outperforms baselines and existing methods (like SteerConf):

• Calibration Error (ECE): Reduced by 4–7× compared to unsteered verbalized confidence.
  • Example (Mistral-7B): ECE dropped from 35.1 (unsteered) to 3.3 (steered).
• Brier Score & MAE: Substantial improvements across all evaluated models and datasets.
• Comparison: The method outperforms prompt-engineering strategies (SteerConf) and logit baselines, proving that reading internal signals is more effective than external manipulation.

5. Significance

• Paradigm Shift: The paper reframes verbalized miscalibration not as a lack of knowledge or a need for retraining, but as a readout failure. The signal exists; the model just fails to use it.
• Mechanistic Insight: It reveals a critical geometric property of LLMs: internal accuracy and verbalized confidence are orthogonal, and joint reasoning actively corrupts this relationship.
• Practical Intervention: The proposed two-stage pipeline offers a lightweight, inference-time solution to improve model trustworthiness without fine-tuning or complex prompt engineering.
• Broader Implication: This work suggests a general pattern for AI safety: when outputs are misaligned with internal representations, the most direct remedy is to identify the internal signal and correct the readout mechanism.