Directional Concentration Uncertainty: A representational approach to uncertainty quantification for generative models

1. Problem Statement

Generative models, particularly Large Language Models (LLMs), lack reliability and transparency in sensitive domains. While Uncertainty Quantification (UQ) is essential for detecting model unreliability, existing methods face significant limitations:

• Rigidity: Many current approaches rely on rigid heuristics that do not generalize across different tasks or modalities.
• Semantic Entropy (SE) Limitations: The state-of-the-art method, Semantic Entropy, relies on clustering generated outputs based on semantic equivalence (using entailment models). This works well for short, structured question-answering (QA) but fails in complex, open-ended, or multi-modal tasks because defining "semantic equivalence" for long or diverse texts is ad hoc and difficult.
• Need for Generalization: There is a need for a UQ framework that is flexible, task-agnostic, and capable of handling multi-modal data (e.g., text + images) without requiring specialized semantic matching models.

2. Methodology: Directional Concentration Uncertainty (DCU)

The authors propose Directional Concentration Uncertainty (DCU), a geometric framework that quantifies uncertainty by measuring the dispersion of model outputs in a continuous embedding space, rather than clustering them semantically.

Core Workflow:

1. Sampling: For a given input $x$, the model generates $N$ independent outputs $\{y_1, y_2, ..., y_N\}$.
2. Embedding: Each output is mapped to a $d$-dimensional vector using a pre-trained embedding model $\phi(\cdot)$.
3. Normalization: The embeddings are $\ell_2$-normalized to lie on a unit hypersphere ($S^{d-1}$):
   $$z_i = \frac{\phi(y_i)}{\|\phi(y_i)\|}$$
4. Statistical Modeling (vMF): The set of normalized embeddings is modeled using the von Mises-Fisher (vMF) distribution, which is defined on a hypersphere. The distribution is parameterized by:
   • Mean Direction ($\mu$): The average direction of the embeddings.
   • Concentration Parameter ($\kappa$): Quantifies how tightly the samples cluster around the mean direction.
5. Uncertainty Calculation:
   • The concentration parameter $\kappa$ is estimated via Maximum Likelihood Estimation (MLE).
   • High $\kappa$ indicates tight clustering (low uncertainty/high consistency).
   • Low $\kappa$ indicates high dispersion (high uncertainty/inconsistency).
   • The final uncertainty score is defined as the inverse concentration: $\kappa^{-1}$.

Differentiation from Semantic Entropy (SE):

• SE: Requires pairwise semantic comparisons (e.g., using NLI models) to group answers into clusters, then computes entropy over cluster probabilities.
• DCU: Operates directly on continuous embeddings. It requires no discrete cluster assignments, no entailment models, and no task-specific definitions of semantic equivalence.

3. Key Contributions

• Novel Framework: Introduction of DCU, a geometric UQ method that replaces semantic clustering with directional concentration analysis in embedding space.
• Modality Agnosticism: By relying on embeddings rather than text semantics, DCU can be applied to any modality (text, images, multi-modal) provided a suitable encoder exists.
• Superior Generalization: Demonstrated that DCU matches or exceeds SE performance on standard QA tasks and significantly outperforms SE on complex, multi-modal tasks where semantic clustering fails.
• Theoretical Insight: The paper argues that the success of sampling-based methods like SE is fundamentally driven by the consistency of the model's output distribution, which can be captured more efficiently via geometric concentration than semantic clustering.

4. Experimental Results

The authors evaluated DCU against Semantic Entropy (SE) using LLaMA-2, Falcon, Mistral (text-only) and LLaVA, Gemma 3 (multi-modal) across several datasets.

A. Text-Only QA Tasks (TriviaQA, SQuAD, NQ-Open)

• Performance: DCU achieved AUROC scores comparable to SE. In several cases (e.g., Falcon on NQ-Open), DCU matched or slightly exceeded SE.
• Stability: Both methods showed low variance in bootstrap resampling, indicating robust uncertainty signals.
• Conclusion: DCU successfully captures the same uncertainty signals as SE without the computational overhead of semantic clustering.

B. Visual Question Answering (ScienceQA)

• Challenge: This task involves multi-modal inputs (text + images) and is more complex than standard QA.
• SE Performance: SE performed poorly (AUROC ~0.51 for LLaVA), indicating it failed to distinguish correct from incorrect answers. The reliance on textual entailment models broke down in the multi-modal context.
• DCU Performance: DCU significantly outperformed SE (AUROC 0.67 for LLaVA vs. 0.51 for SE).
• Significance: This proves that the representational approach generalizes to complex, multi-modal domains where heuristic semantic clustering fails.

5. Significance and Future Implications

• Breaking Modality Barriers: DCU removes the dependency on "semantic equivalence," which is ill-defined for non-text or complex generative tasks. This makes it a viable candidate for UQ in agentic systems and multi-modal pipelines.
• Efficiency: While DCU still requires multiple sampling runs (a cost shared with SE), it eliminates the need for expensive pairwise entailment checks, potentially reducing computational overhead in complex scenarios.
• Future Directions:
  • Application to arbitrary modalities (image generation, audio).
  • Integration into agentic workflows to detect and limit uncertainty propagation before it affects downstream decisions.
• Limitations: The method still requires repeated sampling from the generative model, which can be computationally expensive in real-time applications. However, the authors note that the encoder cost is lower than the generative model cost.

Conclusion

The paper establishes that Directional Concentration Uncertainty (DCU) is a robust, flexible, and generalizable alternative to Semantic Entropy. By shifting the focus from discrete semantic clustering to continuous geometric concentration, DCU provides a unified framework for uncertainty quantification that scales effectively from simple text QA to complex multi-modal reasoning tasks.