Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errors

Technical Summary: Stable Miscalibration in Large Language Models

Problem Statement
High-confidence errors (hallucinations) in Large Language Models (LLMs) are frequently interpreted as evidence of "fragile internal inference"—the idea that a small perturbation to the prompt or context causes the model to flip its answer. This paper challenges that singular view by proposing stable miscalibration: a state where a model produces a confident, incorrect answer that remains locally stable under small perturbations. The author argues that if high-confidence errors are stable rather than fragile, they may not be distinguishable from correct high-confidence answers via sensitivity analysis alone, yet they still pose significant risks for decision-making systems. The core question is: what can be inferred about these errors using measurable confidence responses and small internal probes, given that full internal mechanistic explanations are often inaccessible?

Methodology
The study employs a multi-faceted approach combining a labeled audit set, output-level diagnostics, and internal sensitivity probing across three open-weight models (Llama-3.1-8B, DeepSeek-R1, and Qwen2.5-7B).

1. Audit Dataset: A frozen set of 532 binary factual items across 11 domains (e.g., medical epidemiology, social stats, history) was used. Items were evaluated under three distinct policies:

   • C0 (Forced-Answer): The model must answer "Yes" or "No."
   • C1 (Cautious-Abstention): The model may abstain if the item is under-specified.
   • C2 (Self-Critical Abstention): The model performs an explicit self-critique check before deciding, potentially abstaining.

2. Output-Level Diagnostic ($H_{proxy}$): The author introduces a label-aware audit score designed to rank domains by risk. This score combines two signals:

   • Confidence Variation: The standard deviation of reported confidence ($P(\text{correct})$) across the three policies.
   • Overconfident-Wrong (OCW) Mass: A binary indicator where the C0 baseline is wrong but reports confidence > 0.8.
   • The score is defined as $H_{proxy}(d) = \frac{1}{|D_d|} \sum_{i \in D_d} [\text{Std}_{c \in \{C0,C1,C2\}}(p_{i,c}) + \lambda \cdot \text{OCW}(i)]$.
   • This metric is explicitly labeled as a retrospective audit diagnostic, not a deployment-time estimator for unlabeled data.

3. Internal Sensitivity Probe: To test the "fragility" hypothesis, the author injected Gaussian noise ($\sigma=0.01$) into the embedding space of the three open-weight models. They measured the movement of the final-token hidden states ($S_\ell$) across layers. This was performed on items pre-classified by the C0 audit logs as either Confidently Correct (CC) or Overconfidently Wrong (OCW). The study also tested the effect of self-critical prompting on this sensitivity.

Key Contributions

1. Theoretical Abstraction: The paper posits "stable miscalibration" as a valid uncertainty hypothesis. Using a linear-Gaussian closed-loop abstraction, it demonstrates that local stability (damped perturbations) and correctness need not coincide; a system can converge to a stable state that is systematically wrong.
2. Audit Score for Intervention: The author proposes $H_{proxy}$ as a practical tool to identify domains where abstention-aware self-critique (C2) is most likely to reduce policy-aware loss. The score correlates with where intervention helps, distinguishing domains that appear safe on average but are risky under policy shifts.
3. Internal vs. Output Discrepancy: The study provides empirical evidence that while self-critical prompting induces local stabilization (reducing hidden-state sensitivity across all layers), it does not create a clear sensitivity gap between overconfidently wrong items and confidently correct items.

Results

• Output-Level Findings: The $H_{proxy}$ score shows a strong positive correlation (Spearman $\rho = 0.71$) with the gain achieved by the self-critical policy (C2) over the forced-answer baseline (C0). Domains like medical epidemiology and social stats showed the largest improvements with C2, while entertainment and literature domains showed little to no benefit. However, direct labeled baselines (C0 error rate and Brier risk) ranked the domains even more strongly, confirming that $H_{proxy}$ is a structured diagnostic rather than a superior predictor of raw performance.
• Internal Findings:
  • Stabilization: Self-critical prompting consistently reduced layer-wise hidden-state sensitivity across all three models, supporting the view that such prompts induce local stabilization.
  • No Fragility Gap: Crucially, the difference in sensitivity between the OCW and CC groups was negligible (standardized gap $\approx 0.13$ pooled standard deviations or less, with confidence intervals including zero). This suggests that audit-defined overconfident errors are not uniquely fragile compared to correct answers under the tested perturbations.
  • Robustness: Denser noise sweeps and semantic rewrite checks confirmed that the lack of a fragility gap was not an artifact of the specific perturbation settings.

Significance and Claims
The paper concludes with a modest, practical lesson rather than a full mechanistic reduction. The primary significance is that uncertainty-related behavior relevant to decisions can be studied through confidence responses even when internal mechanisms remain partially understood.

• Stable Miscalibration as a Risk: The findings suggest that high-confidence errors may be "stable and miscalibrated" rather than "fragile." A system can appear robust because its answers are locally stable, yet still produce confident wrong answers in high-impact domains.
• Role of Diagnostics: The proposed audit score is intended for periodic labeled evaluation to triage risks and allocate interventions (e.g., abstention, retrieval, human review) where overconfident-error mass is concentrated. It is not designed as a live safety filter or a label-free estimator.
• Limitations: The author acknowledges that the study uses a frozen, author-curated audit set, which may not generalize to larger datasets or open-ended generation. The "no clear gap" result is specific to the current probe and does not rule out instability in other contexts, but it challenges the assumption that fragility is the primary signature of high-confidence hallucinations.

In summary, the work shifts the focus from viewing high-confidence errors solely as signs of instability to recognizing them as potentially stable miscalibrations, necessitating audit strategies that look beyond simple sensitivity analysis.