Integrating Local and Global Entropy for Uncertainty Quantification in LLMs

Technical Summary: Integrating Local and Global Entropy for Uncertainty Quantification in LLMs

Problem Statement

Large Language Models (LLMs) frequently exhibit "confident hallucinations," producing fluent, specific, yet factually incorrect responses. Existing Uncertainty Quantification (UQ) methods predominantly rely on token-level signals (e.g., log-probabilities or sampling consistency). While effective for detecting local ambiguity, these methods often fail to identify the "confident-but-wrong" failure mode, where a model generates incorrect answers with high token-level confidence. Furthermore, many existing approaches impose practical barriers, such as requiring multiple generation passes, task-specific supervision, sensitivity to response length, or specific architectural assumptions. There is a need for a UQ method that is unsupervised, computable in a single forward pass, length-normalized, and capable of capturing the geometric structure of the model's internal trajectory.

Methodology

The paper proposes Global-Local Uncertainty (GLU), a framework that integrates two distinct uncertainty signals derived from different representational layers of the LLM:

1. Local Uncertainty (Token-Level):

   • Derived from the unembedding layer (next-token prediction).
   • Based on the evidential deep learning framework, decomposing uncertainty into Aleatoric Uncertainty (AU) (distributional ambiguity) and Epistemic Uncertainty (EU) (lack of evidence).
   • AU is calculated as the Shannon entropy of a truncated predictive distribution over top-$k$ tokens.
   • EU is estimated by mapping logits to Dirichlet concentration parameters (using a softplus transformation) and calculating the inverse Dirichlet strength.
   • The local score ($\bar{R}$) is the average of the multiplicative interaction of AU and EU across the $m$ most uncertain tokens in the response.

2. Global Uncertainty (Geometric Complexity):

   • Derived from the embedding layer (hidden-state space).
   • Based on the hypothesis that reliable fact retrieval corresponds to hidden states moving in a tight, consistent direction, while hallucinations correspond to wandering through loosely related directions.
   • The method constructs a Gram matrix ($K = HH^\top$) from the hidden-state trajectory $H$ across all generation steps.
   • Global uncertainty is quantified as the Rényi-2 entropy (collision entropy) of the normalized eigenvalue spectrum of $K$. A high entropy indicates a diffuse, incoherent trajectory (high uncertainty), while low entropy indicates a coherent, low-rank trajectory.
   • To ensure comparability across response lengths and models, the entropy is normalized by $\log T$ and averaged across all transformer layers.

3. Fusion Strategy (GLU):

   • The two signals are fused via a multiplicative gate: $GLU = (1 + \tilde{S}) \cdot \bar{R}$, where $\tilde{S}$ is the normalized global geometric entropy and $\bar{R}$ is the aggregated local uncertainty.
   • This multiplicative design acts as a geometric amplifier: if the trajectory is diffuse (high global uncertainty), the local uncertainty score is up-weighted. This specifically targets the "confident-but-wrong" regime where token entropy is low but geometric coherence is absent.
   • The method requires only a single forward pass, no labeled data, and is architecture-agnostic.

Key Contributions

1. Multi-View Uncertainty Extraction: The paper extracts complementary signals from two distinct layers: global geometric entropy from hidden-state trajectories and local token-level entropy from output logits.
2. Statistical Orthogonality: Empirical analysis demonstrates that global and local signals are statistically near-orthogonal. The global signal recovers the "confident-but-wrong" failure mode that local signals systematically miss.
3. GLU Framework: Introduction of a lightweight, unsupervised score that fuses these signals multiplicatively, avoiding the need for retraining, pseudo-labels, or multiple sampling passes.
4. Comprehensive Validation: Extensive evaluation across three model families (Qwen2.5, Gemma3, Fanar1) and six diverse benchmarks (including factuality, math, and Arabic QA), alongside rigorous ablation studies.

Experimental Results

• Performance: Across 18 model-dataset settings, GLU achieved the best AUROC (Area Under the Receiver Operating Characteristic) in 13 cases and the best Prediction Rejection Ratio (PRR) in 12 cases. It consistently outperformed both local-only baselines (e.g., LogTokU) and global-only baselines (e.g., RAUQ).
• Ablation Studies:
  • Multiplicative vs. Additive: The multiplicative fusion strategy was critical. An additive variant ($Add\text{-}\tilde{S}$) dropped significantly in performance (ranking 13th out of 14 methods), confirming that the global signal must act as a modulator rather than a fixed offset.
  • Robustness: The performance gain was robust across different choices of local and global estimators, suggesting the complementarity is a structural property of LLMs rather than an artifact of specific hyperparameters.
• Calibration: On the TriviaQA benchmark, GLU demonstrated a monotonic relationship between uncertainty scores and empirical error rates, effectively separating correct from incorrect responses across confidence deciles.

Significance and Claims

The paper claims that the structural distinction between the embedding (semantic direction) and unembedding (token probability) layers provides a principled basis for why token-level and geometric signals measure different aspects of uncertainty. By integrating these views, GLU offers a practical, unsupervised solution for reliability prediction that addresses a critical blind spot in current UQ methods: the confident hallucination. The authors assert that GLU matches or outperforms all unsupervised baselines while remaining length-normalized and architecture-agnostic, making it suitable for real-world deployment where computational efficiency and lack of supervision are paramount. The work suggests that the complementarity between local and global uncertainty is a fundamental structural property of LLMs.