💬 NLP

Hallucination Detection via Activations of Open-Weight Proxy Analyzers

This paper introduces a proxy-analyzer framework that detects hallucinations in large language models by analyzing the internal activations of small, locally hosted open-weight models, achieving state-of-the-art performance across diverse analyzer architectures while demonstrating that larger model size does not necessarily correlate with better detection accuracy.

Original authors: Akshita Singh, Prabesh Paudel, Siddhartha Roy

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Akshita Singh, Prabesh Paudel, Siddhartha Roy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart but sometimes unreliable writer (a Large Language Model) who is trying to answer questions based on a specific document. Sometimes, this writer makes things up, or "hallucinates," mixing facts with fiction.

The problem is: How do you catch the writer lying without having to peek inside their brain or ask them to explain their thought process? Usually, if the writer is a closed system (like a big commercial API), you can't see their internal workings.

This paper introduces a clever new solution: The "Proxy Analyzer."

The Core Idea: The "Second Opinion" Detective

Instead of trying to look inside the writer's brain, the authors built a small, independent detective (a smaller AI model) that reads the finished story along with the original source document.

Think of it like this:

The Writer: A student taking a test.
The Source: The textbook.
The Hallucination: The student writing an answer that sounds good but isn't in the textbook.
The Old Way: Trying to look inside the student's head to see if they are guessing. (Impossible if the student is a "black box").
The New Way (This Paper): You hire a small, sharp tutor (the Proxy Analyzer) to read the student's answer and the textbook side-by-side. The tutor doesn't need to know how the student thinks; they just look for tension between the answer and the book.

If the student's answer contradicts the book, the tutor's "brain" (the AI's internal electrical signals) lights up in a specific, predictable way. The system detects these electrical "flares" to spot the lie.

How the Detective Works (The 18 Clues)

The detective doesn't just read the words; it looks at the mechanics of how the AI processes the text. The authors built 18 different "clues" (features) based on how AI transformers work. Here are a few analogies for the most important ones:

The "Attention" Spotlight (Signal 2): Imagine the AI has a spotlight that shines on the source text. When it tells the truth, the spotlight stays on the book. When it lies, the spotlight wanders off or gets confused. The system measures exactly where the light is shining.
The "Memory" vs. "Reading" Battle (Signal 4): AI has two ways to answer: reading the book (good) or relying on its own memorized facts (risky). The system measures which one is winning. If the "memory" is screaming louder than the "reading," it's likely a hallucination.
The "Confusion" Meter (Signal 3): When an AI reads the truth, it spreads its attention out. When it lies, it often gets stuck on a few memorized words, creating a "collapse" in attention. The system spots this collapse.

The Big Experiments

The researchers tested this detective using seven different AI models of varying sizes, ranging from tiny (0.5 billion parameters) to huge (9 billion parameters). They treated these models as the "detectives" to see which one was best at spotting lies.

The Surprising Results:

Bigger isn't always better: Usually, we assume a bigger, more expensive detective is smarter. But here, a 3-billion parameter model actually beat the 8-billion parameter model from the same family. It's like finding that a compact, agile car is faster on a specific track than a massive luxury SUV. The design of the detective mattered more than its size.
The "Small" Detective is Great: A tiny 0.5-billion parameter model (Qwen2.5) performed almost as well as the massive ones. This means you don't need a supercomputer to catch hallucinations; a small, fast, cheap model can do the job just fine.
Beating the Experts: Their system consistently outperformed the previous best method (ReDeEP), which required access to the original writer's internal data. The Proxy Analyzer did it without ever touching the original writer.

Why This Matters

The paper claims this is a game-changer because:

It works on "Black Box" writers: You can use this to check answers from closed systems (like GPT-4) where you can't see the internal code.
It's universal: Since the "lie" is a mismatch between the answer and the source, any AI reading the text will show the same electrical "tension." The detector doesn't care who wrote the answer.
It's efficient: You don't need to run the massive generator twice to check it. You just run a tiny, cheap detector once.

The Catch (Limitations)

The authors admit one weakness: Their detective was trained on short documents (about 1,200 characters). When they tested it on much longer documents (3,000 characters), it got slightly confused because the "length" of the text changed the way the clues looked. They believe they can fix this by retraining the detective on longer texts, but for now, it works best on standard-length documents.

In short: The paper proves you can catch AI hallucinations by hiring a small, independent AI to read the answer and the source, looking for specific electrical "tells" that indicate a lie. And surprisingly, a small, well-designed detective is often better than a giant one.

Technical Summary: Hallucination Detection via Activations of Open-Weight Proxy Analyzers

Problem Statement
Retrieval-Augmented Generation (RAG) pipelines are critical for enterprise applications but suffer from hallucination rates ranging from 5% to 20% on domain-specific tasks. Existing detection methods face significant limitations: post-hoc verification and uncertainty-based approaches are computationally expensive, requiring multiple model passes; white-box mechanistic methods offer high accuracy but require direct access to the generator's internal states. This excludes closed API models (e.g., GPT-4), multi-backend systems, or scenarios where re-running a massive generator solely for detection is cost-prohibitive. The core challenge is detecting hallucinations without accessing the generator's weights or internal activations.

Methodology
The authors propose a proxy-analyzer framework that decouples the detector from the generator. Given a source document ( $S$ ), a question ( $Q$ ), and a candidate answer ( $A$ ), the system feeds this text into a small, locally hosted, open-weight "analyzer" model. The detector relies entirely on the analyzer's internal activations to determine if $A$ is hallucinated, regardless of which model generated $A$ .

The framework extracts 18 features grounded in transformer mechanics during a single forward pass:

Activation Signals (S1–S4):
- S1 (Residual Norm): Tracks the growth of residual stream norms; faithful answers show steady growth, while hallucinations plateau early.
- S2 (Per-Head Source Attention): The most critical feature (~50% of feature importance), measuring attention to the source document per head and layer.
- S3 (Attention Entropy): Faithful reading spreads attention; hallucinations collapse onto memorized tokens.
- S4 (MLP Norm): Tracks the firing strength of parametric memory, analogous to a "Parametric Knowledge Score."
Logit and Interaction Signals (S5–S7): Includes logit-lens trajectories, conditional perplexity, and interaction terms reflecting the dominance of parametric memory over source reading.
External and Lexical Signals (S8–S10): Includes a Vectara HHEM score, answer-to-source length ratio, and Jaccard token overlap.
Window and Slope Signals (S11–S15): Aggregates S1 and S2 over a fixed window of layers and analyzes slopes of residual norms and grounding ratios.
Token-Level Grounding Statistics (S16–S18): New statistics capturing the minimum grounding score, variance, and slope of grounding trajectories per token.

A novel Attention Head Importance (AHI) score is introduced to identify which specific attention heads best separate faithful from hallucinated answers, outperforming raw mean attention signals under out-of-distribution (OOD) conditions.

The system employs a stacking ensemble (Logistic Regression, Random Forest, HistGradientBoosting, XGBoost) trained on 72,135 samples from five datasets (HaluEval, RAGTruth, MedHallu, MiniCheck-Synthetic, ANLI). A specialized classifier, RagtStacking, is trained exclusively on RAGTruth data to capture specific faithfulness failure patterns.

Key Contributions

Feature Framework: Introduction of 18 mechanistically grounded features, including three new token-level grounding statistics, derived from a single forward pass of an open-weight model.
Attention Head Importance (AHI): A supervised metric that identifies specific attention heads critical for grounding, demonstrating superior stability under OOD shifts compared to raw attention means.
Scale vs. Architecture Study: A comprehensive evaluation across seven models (0.5B to 9B parameters) from four families (Qwen2.5, Gemma-2, Pythia, LLaMA-3), revealing that model family and architecture design often outweigh parameter count.
Performance Benchmarking: Demonstration that proxy-analyzers can outperform state-of-the-art white-box methods (ReDeEP) without generator access.
Task-Depth Correlation: Discovery that hallucination detection signals peak at different layer depths depending on the task type (e.g., QA peaks deeper than claim verification).

Results

Primary Benchmark (RAGTruth): Across all seven analyzer architectures, the system consistently outperformed ReDeEP's token-level AUC (0.73) by 7.4 to 10.3 percentage points.
- Qwen2.5-7B achieved an F1 of 0.717 and AUC of 0.834 (with RagtStacking), surpassing ReDeEP's token-level F1 (0.713).
- Gemma-2-9B achieved the highest AUC of 0.836.
- Qwen2.5-0.5B achieved an F1 of 0.706, demonstrating that sub-billion models are viable.
Model Size Convergence: Despite an 18-fold difference in parameter count, the AUC spread across all seven models was only 2.3 percentage points (0.814 to 0.837).
The "LLaMA Anomaly": Contrary to scaling laws, LLaMA-3-3B outperformed LLaMA-3-8B on RAGTruth (AUC 0.822 vs. 0.819; F1 0.698 vs. 0.687). The 8B model only recovered on OOD data, suggesting that within the same family, larger size does not guarantee better detection.
OOD Performance: On LLM-AggreFact, all models clustered within 2.8 percentage points of balanced accuracy. Performance gaps with baselines were attributed to source document length mismatches (1,200 chars in training vs. ~3,000 in AggreFact), not fundamental signal failure. Six of the 18 signals remained stable across length shifts.
Generator Agnosticism: The system performed consistently across generators (GPT-4, GPT-3.5, Mistral-7B, Llama-2 variants), confirming the detection of source-document hallucination patterns rather than generator-specific artifacts.

Significance and Claims
The paper claims that architecture design is a stronger driver of detection quality than raw parameter count. The results challenge the intuition that "bigger is always better," showing that a 0.5B Qwen model can match or beat models ten times its size, and a 3B LLaMA can outperform its 8B counterpart.

The authors assert that their approach provides a universal guard for RAG systems. By separating the detector from the generator, the system can be deployed across diverse backends (including closed APIs) with a single implementation. The findings suggest that hallucination is a property of the mismatch between answer and source, detectable through the internal state of any transformer reading the text, making the specific generator irrelevant to the detection process. The paper concludes that while current OOD gaps exist due to training data constraints (source length), the core signal stability indicates the approach is fundamentally sound for broader deployment.