💬 NLP

Geometry-Calibrated Conformal Abstention for Language Models

This paper proposes Geometry-Calibrated Conformal Abstention, a post-hoc framework that leverages representation geometry to calibrate prediction confidence and enable language models to selectively abstain from answering queries with finite-sample guarantees on both participation rates and response correctness, thereby effectively mitigating hallucinations without requiring retraining.

Original authors: Rui Xu, Yi Chen, Sihong Xie, Hui Xiong

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Xu, Yi Chen, Sihong Xie, Hui Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read friend (a Large Language Model) who loves to chat. Sometimes, this friend knows the answer to your question instantly. Other times, they are completely clueless but, because they are so eager to please, they make up a plausible-sounding story instead of admitting they don't know. This is called "hallucination."

The paper you shared proposes a new way to teach this friend when to say, "I don't know," without having to retrain their entire brain. They call this system Conformal Abstention (CA).

Here is how it works, broken down into simple concepts and analogies:

1. The Problem: The "Guessing Game"

Currently, if you ask your AI friend a question it doesn't know, it feels pressure to give an answer. In school, if you guess on a multiple-choice test, you might get lucky and get a point. If you write "I don't know," you get zero. So, the AI learns that guessing is always better than admitting ignorance. This leads to confident-sounding lies.

2. The Solution: A "Confidence Check"

The authors built a post-hoc (after-the-fact) system that acts like a security guard at the door of the AI's answers. Before the AI's answer is shown to you, this guard checks: "Is the AI actually confident and knowledgeable about this, or is it just bluffing?"

If the AI is bluffing, the guard stops the answer and says, "I don't know." If the AI is truly confident, the answer goes through.

3. The Secret Sauce: "Geometry Signals"

How does the guard know if the AI is bluffing? It doesn't just look at the final words; it looks inside the AI's brain while it's thinking.

Think of the AI's brain as a factory with many assembly lines (layers). To make an answer, the AI moves a piece of information (a "token") down the line.

The Knowledge Injection (MLP): There is a specific machine in the factory called the MLP (Multi-Layer Perceptron). Think of this as the Library where the AI stores its facts.
The Geometry Check: The new system watches how the "Library" machine changes the information as it passes through. It measures three things using geometry (shapes and angles):
1. Proximity (How close is the change?): Did the Library machine actually do something to the information, or did the information just pass through unchanged? If the Library touched it, that's a good sign.
2. Rotation (Did the direction change?): Did the Library turn the information in a new direction? If the information spins wildly, it might mean the AI is confused. If it spins smoothly toward a known fact, that's good.
3. Alignment (Is it on the right path?): Imagine all the AI's "good" knowledge lives inside a specific cone-shaped tunnel. If the information stays inside this tunnel, it's likely a correct, reliable fact. If it wanders outside the tunnel, it's likely a hallucination.

The Analogy:
Imagine you are asking a tour guide about a city.

Good Answer: The guide points to a specific landmark, turns their body toward it, and walks confidently down the main street (High proximity, good rotation, aligned with the "tourist path").
Bad Answer (Hallucination): The guide waves their hands vaguely, spins in circles, and points toward a field that doesn't exist (Low proximity, chaotic rotation, wandering outside the "tourist path").

The system uses these geometric "body language" signals to decide if the answer is trustworthy.

4. The "Guarantee" (The Safety Net)

The paper uses a mathematical method called Conformal Prediction. Think of this as a statistical promise.

The system doesn't just guess; it calculates a threshold. It promises: "If we only let answers through that pass our geometric check, we guarantee that 75% of the answers you see will be correct."

It also guarantees that the AI won't be too shy. It promises: "We won't say 'I don't know' so often that we stop answering 90% of the time." It balances being safe with being helpful.

5. The Results

The authors tested this on six different types of questions (from simple facts to complex reasoning). They found that their geometric "body language" check was much better at spotting lies than previous methods.

Old methods were like checking if the AI sounded confident (which liars can do).
Their method checks the AI's internal "body language" to see if it actually knows the answer.

In summary: This paper gives Large Language Models a new "lie detector" built into their own internal geometry. It allows them to admit ignorance when they are unsure, ensuring that when they do speak, they are much more likely to be telling the truth.

1. Problem Statement

Large Language Models (LLMs) often fail to recognize their own knowledge limits. When faced with queries outside their training distribution or knowledge base, they tend to generate plausible-sounding but factually incorrect responses (hallucinations) rather than abstaining (saying "I don't know").

Training Limitations: Retraining models to reward abstention is difficult due to the scarcity of benchmarks that explicitly penalize guessing and reward ignorance. Furthermore, such interventions can lead to overfitting or overly conservative behavior where models refuse to answer even when they know the answer.
Existing Post-hoc Methods: Traditional Conformal Prediction (CP) offers finite-sample guarantees for uncertainty but relies on "non-conformity scores." In open-ended text generation, defining a ground-truth score function is intractable because correct answers can take infinite semantically equivalent forms.
Core Question: Can LLMs be equipped with a post-hoc mechanism to reliably abstain from answering when uncertain, without retraining, while providing theoretical guarantees on correctness?

2. Methodology: Conformal Abstention (CA)

The authors propose Conformal Abstention (CA), a framework that combines Conformal Prediction theory with a novel representation geometry-based calibration strategy.

A. Theoretical Framework (Conformal Abstention)

CA adapts CP to the language generation setting by focusing on prediction confidence rather than non-conformity scores. It establishes two finite-sample guarantees:

Participation Guarantee: Bounds the probability that the model provides an answer (does not abstain).
Conditional Correctness Guarantee: Bounds the probability that a provided answer is correct, given that the model chose to participate.

The framework uses an uncertainty score $r(\hat{y}|x)$ (e.g., perplexity). By setting a threshold $\tau$ based on a calibration set, the model abstains if the score exceeds $\tau$ . Theoretical proofs (Theorem 3.1, Lemma 3.2) demonstrate that if the input-output pairs are exchangeable, the conditional correctness of retained answers is bounded below by a user-specified confidence level.

B. Calibration via Representation Geometry

To ensure the uncertainty score faithfully reflects the model's internal state (knowledge vs. ignorance), the authors introduce a calibration strategy based on the geometry of token representations across transformer layers. They hypothesize that when a model leverages internal knowledge, the MLP (Multi-Layer Perceptron) updates induce specific geometric transformations in the hidden states.

They extract three geometric signals from the trajectory of the last query token's embedding:

Proximity ( $\omega$ ): Measures the contribution of the MLP update. It calculates the distance between the pre-MLP and post-MLP embeddings. A small distance (high proximity) indicates the MLP actively shaped the representation.
Embedding Rotation ( $\theta$ ): Measures the angle of rotation induced by the MLP update. Large rotations suggest a significant shift in the prediction direction, often associated with retrieving specific knowledge.
Alignment ( $\phi$ ): Measures the angle between the token embedding and the dominant anisotropic direction of in-distribution data. High alignment suggests the model is operating within its learned distribution; deviation suggests extrapolation (uncertainty).

Hypotheses Validated:

H1: Correct predictions are associated with strong knowledge contribution (high $\omega$ ) inducing significant rotation (high $\theta$ ).
H2: Correct predictions remain aligned with the in-distribution anisotropic cone (low $\phi$ ), while incorrect ones deviate.

C. Calibration Pipeline

Feature Extraction: For each token, the model computes a feature vector $(\Omega, \Theta, \Phi_{in}, \Phi_{out})$ capturing the geometric signals across all layers.
Distance Calculation: The system computes the Mahalanobis distance of these features from the distributions of correct and incorrect predictions.
Calibrator Training: An XGBoost model is trained to map these geometric features (combined with raw perplexity) into a refined uncertainty score.
Decision: The final uncertainty score determines whether to abstain based on the conformal threshold.

3. Key Contributions

Conformal Abstention Framework: A post-hoc method that provides rigorous finite-sample guarantees for both participation rates and conditional correctness in open-ended generation, bypassing the intractability of non-conformity scores.
Geometry-Based Calibration: A novel approach to uncertainty quantification that leverages the internal geometric dynamics of transformer layers (MLP contributions, rotations, and anisotropy) to better align confidence with actual correctness.
Theoretical Guarantees: Proofs showing that exchangeability is preserved under the deterministic transformations used to derive uncertainty scores, ensuring the validity of the conformal bounds.
Comprehensive Evaluation: Extensive experiments across six diverse datasets (NQ, TruthfulQA, GSM8K, etc.) and multiple model sizes (Gemma-3, LLaMA-3).

4. Experimental Results

Performance: The proposed method significantly outperforms strong baselines (including Perplexity, Semantic Entropy, Attention-based methods, and verbalized confidence) across all datasets.
Conditional Correctness: Under conformal abstention, the method achieves an average conditional correctness of 75.0%. This means that when the model decides to answer, it is correct 75% of the time, a significant improvement over baselines which often hover around 50-60%.
Robustness: The method maintains high performance across different participation rates (from 10% to 90% of queries answered).
Ablation Studies: Permuting the geometric signals ( $\omega, \theta, \phi$ ) causes comparable performance drops, indicating that all three signals are crucial and complementary.
Baseline Calibration: The authors show that simply applying their calibrator to existing baselines often degrades performance (due to overfitting on low-dimensional signals), whereas their high-dimensional geometric features benefit significantly from calibration.

5. Significance

This work addresses a critical reliability gap in LLM deployment. By enabling models to selectively abstain with theoretical guarantees, CA reduces the risk of hallucinations in high-stakes domains (healthcare, law, science) without requiring expensive retraining. The introduction of representation geometry as a proxy for "knowledge involvement" offers a new paradigm for uncertainty quantification, moving beyond surface-level probabilities to the internal mechanics of how models process information. This makes the system more trustworthy and actionable for real-world applications where admitting ignorance is preferable to providing false information.