🤖 machine learning

Hallucination Is Linearly Decodable from Mid-Layer Hidden States in Quantized LLMs

This paper demonstrates that a linearly separable truthfulness signal encoding hallucination is strongly present in the mid-layer hidden states of 4-bit quantized open-source LLMs, enabling detection with near-perfect AUROC scores using simple linear probes, whereas sampling-based methods perform significantly worse under the same evaluation protocol.

Original authors: Aizierjiang Aiersilan

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aizierjiang Aiersilan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly unreliable, robot assistant. Sometimes it tells you the truth, and sometimes it makes up facts that sound convincing but are completely wrong (these are called "hallucinations").

This paper is like a detective report trying to figure out: How can we tell if the robot is lying, and where in its "brain" is the lie hiding?

The researchers tested this on three popular, open-source robot brains (Llama, Mistral, and Qwen) that have been shrunk down to fit on a standard home computer (using a technique called "4-bit quantization," which is like compressing a high-definition movie into a small file without losing too much quality).

Here is what they found, explained simply:

1. The "Truth Detector" is a Simple Line

The researchers tried two main ways to catch the robot lying:

The "Gambler" Approach (Sampling): Ask the robot the same question 10 times. If it gives you 10 different answers, maybe it's confused or lying. If it gives the same answer every time, maybe it's telling the truth.
The "X-Ray" Approach (Probing): Look directly inside the robot's brain while it's thinking. Specifically, they looked at the "hidden states" (the electrical signals between layers of the brain) to see if there is a pattern that says "Truth" or "Lie."

The Result: The "X-Ray" approach was a massive winner. By looking at just one specific layer in the middle of the robot's brain, a very simple math tool (a "linear probe") could tell if the robot was lying with 90% to 100% accuracy.

The "Gambler" approach (asking it 10 times) was basically useless in this specific test, performing no better than flipping a coin (around 50% accuracy).

2. Where in the Brain is the Lie?

The researchers found that the "truth signal" isn't everywhere. It's like a specific room in a skyscraper where the truth is written on the wall.

For the Llama and Mistral robots (which have 32 floors), the truth is most visible on floors 13 to 18.
For the Qwen robot (which has 28 floors), the truth is most visible on floors 19 to 25.

If you check the robot's brain at these specific "floors," you can almost instantly know if it's being honest.

3. Why the "Gambler" Approach Failed

You might wonder, "Why didn't asking the robot 10 times work?"
The paper explains this with a clever distinction:

The Probe (X-Ray) was given a specific answer (e.g., "The capital of France is Paris") and asked, "Does your brain recognize this as true?" It looked at the robot's internal reaction to that specific sentence.
The Gambler (Sampling) was just asked a question and told to "write 10 different stories." It measured how confused the robot was about the question, not whether a specific answer was true.

Because the test was set up to check specific answers, the "Gambler" was looking in the wrong place. It was like trying to find a specific typo in a document by asking the author to write the document 10 times from scratch, rather than just reading the document once.

4. The "Simple Line" vs. The "Complex Machine"

The researchers tried using a super-complex math tool (an MLP) to find the lies, thinking it might be smarter than a simple tool.
The Surprise: The complex tool didn't do much better than the simple tool. The "truth" in the robot's brain is actually very straightforward—it's almost like a straight line. You don't need a super-computer to find it; a simple ruler works just fine.

5. The "Attention" Clue

There was one other trick that worked well, but only when the robot was reading a specific text (like a Wikipedia article).
They looked at the robot's "attention" (what it's focusing on) at the very first step.

If the robot was telling the truth based on the text, its attention was focused and calm.
If it was hallucinating, its attention was scattered and confused.
This gave a very strong clue (up to 94% accuracy) without needing any extra math or time.

The Big Takeaway

If you want to catch a small, compressed AI robot lying about facts:

Don't ask it the same question 10 times (it wastes time and doesn't work well here).
Do peek inside its brain at the middle layers (around the 15th or 20th layer).
You can use a very simple, fast math check to spot the lie with near-perfect accuracy.

The paper concludes that for these specific types of tests, looking inside the robot's brain is much faster and more accurate than asking it to repeat itself.

Technical Summary: Hallucination Is Linearly Decodable from Mid-Layer Hidden States in Quantized LLMs

Problem Statement
Hallucinations—fluent but factually unsupported generations—remain a critical failure mode for Large Language Models (LLMs) in real-world deployments. Detecting these errors at inference time is a practical necessity. Existing detection approaches generally fall into two categories:

Sampling-based methods: These generate multiple stochastic continuations for a prompt and flag hallucinations based on lexical, semantic, or internal-state disagreement (e.g., Self-Consistency, Semantic Entropy, INSIDE). These methods are computationally expensive, requiring $K \ge 5$ extra forward passes per prompt.
Representation-based methods: These train small classifiers (probes) to decode truthfulness directly from the model's intermediate hidden states. These require only a single forward pass but are rarely compared head-to-head with sampling methods on identical models, particularly in the resource-constrained, quantized regime used by practitioners.

Methodology
The authors propose a unified evaluation framework designed to run end-to-end on a single consumer-grade GPU (8 GB VRAM) using 4-bit NF4 quantization.

Models: Three instruction-tuned 7B–8B models were evaluated: Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Qwen2.5-7B-Instruct.
Datasets: Four benchmarks were used, each providing paired truthful (label=1) and hallucinated (label=0) answers for the same prompt to ensure near-perfect 50:50 class balance:
- TruthfulQA: Multiple-choice questions with plausible but incorrect distractors.
- HaluEval-QA: Knowledge-conditioned QA with explicit evidence passages.
- FEVER: Wikipedia-supported claims labeled as supported or refuted.
- Synthetic: Controlled pairs of true/false statements about world facts.
Detection Protocols: Four methods were evaluated under a paired-label protocol, where the candidate answer is supplied to the model rather than generated:
1. Linear and MLP Probes (SAPLMA-style): Classifiers trained on per-layer hidden states (last-token pooling) to predict truthfulness.
2. INSIDE EigenScore: A sampling-based method measuring the log-determinant of the covariance of hidden states across $K=10$ stochastic completions.
3. Self-Consistency: Measuring agreement among $K=5$ completions via exact-match plurality and sentence-embedding similarity.
4. Attention Entropy: Summarizing the Shannon entropy of the last-token attention distribution at specific transformer blocks.

Key Results
The study yields three primary findings regarding the performance and nature of these detectors:

Superiority of Probes in Paired-Label Settings: Linear probes on single mid-network layers achieve AUROC scores between 0.904 and 1.000 across all models and datasets. In contrast, sampling-based methods (INSIDE and Self-Consistency) perform near chance, with AUROC scores never exceeding 0.541.
Linearity of the Truthfulness Signal: The signal is approximately linear. MLP probes rarely outperform linear probes by more than 0.01 AUROC, suggesting that the truthfulness direction is a linear subspace within the hidden state representation.
Consistent Peak Probing Layers: The "truthfulness signal" peaks in a consistent band of layers across model families:
- Llama & Mistral (32 layers): Blocks 13–18.
- Qwen (28 layers): Blocks 19–25.
- On natural-language benchmarks, the peak is consistently in the second half of the network.
Attention Entropy as a Complementary Signal: In knowledge-grounded settings (HaluEval-QA), the attention entropy of the first transformer block provides a strong signal (AUROC 0.866–0.941) at no additional inference cost. This signal does not generalize to datasets lacking explicit evidence passages.
Quantization Robustness: High detection performance is maintained under 4-bit NF4 quantization, with peak AUROC reaching 0.998 on HaluEval-QA and 1.000 on the synthetic benchmark.

Significance and Claims
The paper claims that the observed weakness of sampling-based methods is not an inherent limitation of those techniques but a structural mismatch between the evaluation protocol and the information accessed:

The Mismatch: In the paired-label protocol, probes observe the model's internal representation of a supplied answer, allowing them to detect if the model internally distinguishes that answer as truthful. Conversely, sampling methods measure the model's generative uncertainty about the prompt alone, which is weakly correlated with the truthfulness of a specific, pre-supplied candidate answer.
Practical Implication: For white-box detection within a single deployed model, probing hidden states is both more effective and significantly less expensive (single forward pass) than sampling-based approaches.
Deployment: The results suggest that linear probes can be deployed on consumer hardware alongside quantized models without the overhead of de-quantization or multiple generations.

The authors conclude that while sampling methods may recover effectiveness in "generate-then-judge" settings, the linearly decodable nature of hallucination signals in mid-layer hidden states offers a highly efficient alternative for detecting factual errors in instruction-tuned, quantized LLMs.