💬 NLP

Where Does Reasoning Break? Step-Level Hallucination Detection via Hidden-State Transport Geometry

This paper proposes a step-level hallucination detection framework that identifies reasoning errors as localized excursions in hidden-state transport geometry, utilizing a contrastive PCA-based teacher and a distilled BiLSTM student to outperform existing baselines in localizing the first error during single-pass inference.

Original authors: Tyler Alvarez, Ali Baheri

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tyler Alvarez, Ali Baheri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) solving a complex math problem or writing a story is like a hiker walking along a narrow, well-trodden mountain path. As long as the hiker stays on the path, they are moving "correctly." But sometimes, the hiker takes a wrong turn. Once they step off the path, they might wander around, eventually stumbling back onto a trail that looks similar, but they are now lost. The problem is that by the time they reach the end, they might confidently claim they are at the summit, even though they took a wrong turn miles ago.

This paper, titled "Where Does Reasoning Break?", introduces a new way to catch that exact moment the hiker steps off the path, rather than just checking if they arrived at the wrong destination.

Here is the breakdown of their approach using simple analogies:

1. The Problem: Checking the Whole Trip vs. The Wrong Turn

Most current methods for spotting AI "hallucinations" (making things up) are like a tour guide who only looks at the final photo of the trip. They say, "This photo looks weird; the whole trip was a failure."

The flaw: They can't tell you where the hiker went wrong. Was it step 2? Step 50?
The goal: The authors want to catch the very first step where the hiker leaves the path, so the error can be fixed immediately.

2. The Core Idea: The "Hidden State" Trajectory

The authors treat the AI's internal thinking process (its "hidden states") not as a list of words, but as a movie of movement through a high-dimensional space.

The Stable Manifold: Think of "correct reasoning" as a smooth, invisible highway. When the AI is thinking correctly, its internal "movie" stays glued to this highway.
The Excursion: When the AI makes a mistake, its internal movie suddenly jerks off the highway. It's a "localized excursion"—a sudden, weird jump in the data.
The Insight: Even if the AI later tries to act normal again, that first jump leaves a geometric scar that can be measured.

3. The Two-Part System: The Teacher and the Student

The paper proposes a system with two characters: a Teacher and a Student.

The Teacher (The "Magic Glasses")

How it works: The Teacher is a super-smart diagnostic tool that has a cheat sheet. It knows exactly which steps in the AI's reasoning were right and which were wrong (because it was trained with labels).
The Trick: It uses a special pair of glasses called Contrastive PCA. Imagine looking at a crowded room where everyone is wearing different colored shirts. The Teacher's glasses filter out all the "noise" (like the color of the shirts, the size of the room, or the topic of conversation) and zoom in only on the specific direction where the "wrong turn" happens.
The Result: The Teacher can spot the exact moment the AI leaves the highway with incredible accuracy.
The Catch: The Teacher is useless in the real world because it needs the cheat sheet (the labels) to work. You can't give an AI a cheat sheet while it's actually solving a problem for you.

The Student (The "Field Detective")

How it works: The Student is a lightweight, deployable model designed to work without a cheat sheet. It tries to learn what the Teacher sees, but it only has access to the raw "movie" of the AI's thinking.
The Training: The Student is trained by watching the Teacher. It learns to mimic the Teacher's "instability score."
The Goal: The Student is supposed to be the tool we actually use in real life to catch errors in a single pass.

4. The Big Discovery: The "Margin" Problem

The paper ran many experiments and found a surprising result:

In the Classroom (In-Domain): When the Student is tested on the same type of AI and data it was trained on, it works great. It catches the wrong turns almost as well as the Teacher.
In the Wild (Cross-Model/Dataset): When the Student is tested on a different AI model or a different type of problem, it falls apart. Its performance drops to near-random guessing.

Why?
The authors explain this using the concept of a "Transport Margin."

Imagine the "correct path" is a wide, safe zone, and the "wrong turn" is a cliff.
The Teacher sees a huge, clear gap (a wide margin) between the safe zone and the cliff.
The Student learns to recognize the cliff, but it learns to recognize it based on the specific texture of the ground in the training room.
When you move the Student to a new room (a new AI model), the ground texture changes. The Student gets confused because it was looking for the wrong texture, not the actual distance to the cliff.
The Conclusion: The central obstacle to making this work in the real world isn't finding the error; it's teaching the Student to recognize the size of the gap (the margin) regardless of what the ground looks like.

Summary

This paper reframes AI hallucination detection as a geometry problem. Instead of asking "Is this sentence true?", it asks "Did the AI's internal movement just take a weird jump off the highway?"

They proved mathematically that their "magic glasses" (Contrastive PCA) are the best way to find these jumps. They built a "Student" to do the job in the real world, but discovered that the Student currently fails when the environment changes. The paper concludes that to fix this, we need to teach the Student to preserve the geometric distance (the margin) between right and wrong, rather than just memorizing the specific patterns of the training data.

Technical Summary: Where Does Reasoning Break? Step-Level Hallucination Detection via Hidden-State Transport Geometry

1. Problem Definition

Large Language Models (LLMs) frequently hallucinate during multi-step reasoning tasks, generating fluent but incorrect intermediate steps that propagate errors to the final output. Existing detection methods suffer from three primary limitations:

Trace-Level Aggregation: Most detectors (e.g., semantic entropy, self-consistency, SelfCheckGPT) assign a single confidence score to an entire output, failing to localize the first error.
Inference Overhead: Methods like self-consistency require multiple forward passes and sampling, which is computationally expensive.
Static vs. Dynamic: Current probing techniques often analyze static factual statements rather than the dynamics of an unfolding reasoning trajectory.

The paper posits that detecting the precise moment reasoning breaks (first-error localization) in a single forward pass is a prerequisite for trustworthy deployment. The core challenge is identifying where a reasoning chain deviates from a coherent path without relying on external verifiers, sampling, or inference-time labels.

2. Methodology: GeoReason

The authors propose GeoReason, a framework that treats the sequence of hidden representations in a single forward pass as a trajectory in representation space. Hallucination is framed not as a semantic error, but as a localized excursion in transport cost away from a stable manifold of locally coherent transitions.

The system consists of two components: a Label-Conditioned Teacher and a Deployable Student.

2.1. The Teacher (Diagnostic Upper Bound)

The teacher is a non-deployable model used to quantify the existence of a geometric signal in hidden states. It operates as follows:

Trace Normalization: For a labeled trace, the teacher computes a local coordinate system centered on the "correct prefix" (steps before the first error). It normalizes hidden states relative to the mean and variance of these correct steps.
Contrastive PCA (cPCA) Lens: The teacher learns a projection matrix $U$ $U$ using contrastive PCA. This lens is optimized to maximize the separation between the distribution of correct transitions ( $P_0$ $P_{0}$ ) and first-error transitions ( $P_1$ $P_{1}$ ) while suppressing high-variance directions inherent to correct reasoning.
- The objective maximizes the transport separation: $\Gamma(U) = \text{Tr}(U^\top M U)$ , where $M = (\mu_1 - \mu_0)(\mu_1 - \mu_0)^\top + C_1 - \alpha C_0$ .
Geometric Feature Extraction: In the projected space, the teacher extracts a feature block for each step, including:
- Position: Distance from the correct manifold.
- Motion: Velocity ( $\|\Delta z_t\|$ ) and acceleration ( $\|\Delta^2 z_t\|$ ).
- Directional Persistence: Cosine similarity between consecutive transition vectors.
Scoring: A lightweight MLP maps these geometric features to a step-level hallucination probability.

2.2. The Student (Deployable Detector)

The student is a deployable BiLSTM model designed to operate without inference-time labels, sampling, or external verifiers.

Architecture: It takes raw hidden states as input and outputs step-level probabilities.
Training via Distillation: The student is trained to reproduce the teacher's instability signal using a mixture of:
1. Supervised Loss: Standard binary cross-entropy on step labels.
2. Knowledge Distillation: Kullback-Leibler (KL) divergence between the student's and teacher's probability distributions (soft targets).
3. Feature Distillation: An auxiliary head forces the student to match the teacher's geometric feature representations.
Inference: The student performs a single forward pass. The predicted first error is the first step where the probability exceeds a threshold $\theta$ .

3. Theoretical Contributions

The paper provides three key theoretical guarantees:

Optimality of cPCA (Theorem 3.1): It proves that contrastive PCA is the optimal projection for maximizing the transport separation between first-error states and correct states under a squared 2-Wasserstein distance objective.
Localization Bound (Theorem 3.2): It establishes that single-pass first-error localization is guaranteed if the first error creates a positive "transport margin" over preceding correct transitions. The error need not remain anomalous forever; detection only requires the first wrong step to create a detectable excursion.
Distillation Condition (Proposition 3.3): It reduces the agreement between teacher and student to a margin-preservation condition. If the student preserves the teacher's decision margin under distribution shifts, the first-error index will be identical.

4. Experimental Results

The method was evaluated on ProcessBench, PRM800K, HaluEval, and TruthfulQA.

4.1. In-Domain Performance

Step-Level Detection: Both models outperformed entropy-based, probing-based, and attention-based baselines. The teacher achieved the highest AUROC on three of four datasets (e.g., 96.0 on TruthfulQA). The student achieved the best performance on PRM800K (99.8 AUROC).
First-Error Localization: The teacher and student significantly outperformed baselines in pinpointing the exact step where reasoning broke. For instance, the student achieved 96.8% accuracy on TruthfulQA.

4.2. Generalization and Deployment Obstacles

Cross-Model Transfer: The teacher demonstrated stable performance across different model families (Qwen, Llama, Mistral), confirming that a geometric signal exists in hidden spaces regardless of the specific model.
Student Collapse: The student performed well when tested on the same model it was trained on but collapsed to near-random performance (AUROC ~33–58%) when evaluated on different model families or datasets.
Analysis: The paper attributes this gap to the student's failure to preserve the transport margin under distribution shift. While the teacher captures a mechanistic instability signal intrinsic to the geometry of errors, the student learns a representational signal that absorbs model-specific encoding quirks, failing to generalize when the underlying distribution shifts.

5. Significance and Claims

The paper claims to recast step-level hallucination detection as a problem of trajectory dynamics rather than static semantic verification.

Geometric Framing: It introduces a novel view where hallucinations are "localized excursions" in hidden-state transport cost, distinct from the stable manifold of coherent reasoning.
Diagnostic vs. Deployable: By separating the existence of a signal (teacher) from the ability to recover it without labels (student), the authors identify the central obstacle to deployment: preserving the contrastive transport margin under distribution shift.
Theoretical Rigor: The work provides the first theoretical justification for using contrastive PCA in this context and establishes conditions under which single-pass localization is mathematically guaranteed.

The authors conclude that while the geometric signal for first-error detection is robust and transferable (as shown by the teacher), current distillation methods are insufficient for cross-domain deployment because they fail to preserve the critical margin required for reliable decision boundaries.