Technical Summary: Retrieval-Augmented Linguistic Calibration (RALC)

Problem Statement

Reliable confidence estimation is critical for the trustworthy deployment of Large Language Models (LLMs) in human decision-making. While existing methods represent confidence as scalar probability values (e.g., token-level probabilities or verbalized scores), humans struggle to reason accurately with numerical probabilities. Consequently, linguistic markers (e.g., "likely," "may") are preferred as natural interfaces. However, treating these markers as scalars discards the inherent subjectivity of linguistic interpretation: different readers map the same expression to different perceived probability values. Furthermore, co-occurring linguistic cues and contextual variations pose unique challenges that scalar representations fail to capture. Current post-hoc calibration methods operate entirely in numerical space, lacking mechanisms to propagate calibrated signals back into natural language, while existing linguistic strategies often function as black-box procedures without principled control.

Methodology

1. Distributional Modeling of Linguistic Confidence

The authors model linguistic confidence not as a single scalar, but as a distribution over plausible perceived probability values that a statement is correct.

• Formalism: Confidence is represented as a Beta distribution, $S = \text{Beta}(\alpha, \beta)$.
  • The mean $\mu = \alpha/(\alpha+\beta)$ captures the central tendency of perceived confidence across readers.
  • The concentration $\kappa = \alpha + \beta$ encodes the strength of agreement (variance) among readers.
• Estimation: A confidence estimator $g$ maps a response $R$ to a Beta distribution. This is parameterized by fitting a Beta distribution to scores generated by an ensemble of LLMs acting as a human audience surrogate, capturing the interplay of multiple linguistic cues and context.

2. Faithfulness Divergence (FD)

To evaluate confidence beyond population-level calibration (Expected Calibration Error, ECE), the paper introduces Faithfulness as an instance-level dimension.

• Definition: Faithfulness measures the "surprise" induced in audience beliefs upon the revelation of the ground truth.
• Metric: Faithfulness Divergence (FD) is defined as a concentration-weighted Bayesian updating cost:
  $$FD_i := (\alpha_i + \beta_i) \cdot \text{KL}(S^*_i \parallel S_i)$$
  where $S_i$ is the prior distribution and $S^*_i$ is the posterior updated with the ground-truth label $y_i$.
• Rationale: FD quantifies the information-theoretic surprise, weighting the KL divergence by the concentration (effective sample size). A high-concentration, misaligned belief incurs a higher penalty (surprise) than a low-concentration misalignment, reflecting the intuition that strongly held incorrect beliefs are more surprising.

3. Retrieval-Augmented Linguistic Calibration (RALC)

RALC is a lightweight, post-hoc pipeline that transforms raw LLM responses into calibrated and faithful outputs directly in linguistic space. The pipeline consists of three stages:

1. Signal-Space Calibration: The estimated confidence distribution mean is calibrated using Platt scaling (logistic regression) against binary correctness labels. The concentration parameter is preserved, as it represents the strength of agreement rather than a target for calibration.
2. Lexicon Construction: A pre-built lexicon maps hedging expressions (e.g., "tend to," "certainly") to their corresponding Beta distributions of perceived confidence. These distributions are derived by having an LLM ensemble evaluate rewritten non-verifiable statements containing specific hedging cues.
3. Retrieval-Augmented Rewriting: The calibrated distribution serves as a retrieval signal. The system performs a two-stage K-Nearest Neighbors (KNN) search:
   • Shortlisting: Based on the absolute distance between the calibrated mean and the lexicon entry means.
   • Ranking: Based on the 1-Wasserstein distance to match both the central tendency and the spread (concentration) of the distribution.
   • Rewriting: The top-$k$ retrieved hedging expressions are passed to an LLM editor along with the original response to rewrite the text, ensuring the new response's perceived confidence aligns with the calibrated signal.

The pipeline is compatible with diverse upstream signals, including linguistic confidence, token probability, and semantic uncertainty (derived from self-consistency sampling).

Key Contributions

1. Distributional Framework: Formalizing linguistic confidence as a Beta distribution to capture the interplay of cues and reader interpretation variability, moving beyond discrete mapping and scalar quantification.
2. Faithfulness Metric: Introducing Faithfulness Divergence (FD) as a new, instance-level evaluation dimension that quantifies the information-theoretic surprise of confidence beliefs upon truth revelation, addressing the limitations of scalar metrics like Brier score and NLL in distributional settings.
3. RALC Pipeline: Proposing a generalizable, post-hoc framework that calibrates confidence in linguistic space via retrieval-augmented rewriting, compatible with various confidence estimation signals.

Experimental Results

The framework was evaluated across five LLM families (GPT-OSS-20B, Llama-3.1-8B, Qwen3-8B, Mistral-7B, Gemma-4-31B) and three benchmarks (MMLU, SQuAD 2.0, TruthfulQA).

• Performance Gains: RALC improved in-domain faithfulness by up to 66% and calibration (generalized ECE) by up to 58% compared to pre-calibration baselines.
• Comparison: RALC outperformed both black-box baselines (Hedged QA, which prompts for hedging without calibrated signals) and grey-box baselines (Direct Beta-Guided Rewrite, which lacks structured lexicon retrieval).
• Signal Robustness: Semantic uncertainty emerged as the most robust confidence signal, yielding the strongest improvements in both calibration and faithfulness across all domains.
• Cross-Domain Transfer: The pipeline demonstrated strong transferability; in some cases, cross-domain calibrators outperformed in-domain ones when the target domain had weak miscalibration bias, allowing a source with stronger bias to learn a more decisive correction.
• Metric Validation: Experiments confirmed that FD correctly ranks surprise levels consistent with distributional profiles (penalizing high-concentration misalignments), whereas KL divergence, Brier score, and NLL failed to recover this ordering.

Significance

The paper argues that linguistic confidence is inherently distributional due to the subjectivity of interpretation. By modeling this distribution and introducing Faithfulness as a complementary evaluation dimension, the work provides a principled way to assess how well a model's expressed confidence aligns with reality at the instance level. The RALC pipeline demonstrates that calibrated confidence signals can be effectively propagated back into natural language through retrieval-augmented rewriting, offering a scalable, post-hoc solution that enhances the trustworthiness of LLMs without requiring access to model internals or retraining. The results suggest that structured lexicon retrieval is superior to unconstrained generation for aligning linguistic output with calibrated confidence.