💬 NLP

CORTEX: Token-Level Hallucination Detection in RAG via Comparative Internal Representations

The paper proposes CORTEX, a token-level hallucination detection method for Retrieval-Augmented Generation that identifies ungrounded content by comparing a model's internal representations with and without retrieved documents, while leveraging information propagation and span-consistent smoothing to achieve fine-grained localization of hallucinations.

Original authors: Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto, Daisuke Kamisaka

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kazuaki Furumai, Shuichiro Haruta, Kazunori Matsumoto, Daisuke Kamisaka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read assistant (the AI) who is trying to answer a question for you. To make sure the answer is accurate, you give the assistant a stack of reference books (the retrieved documents) to look at while they write.

Sometimes, the assistant does a great job, sticking strictly to the books. Other times, they might accidentally invent a fact or mix up a detail, even though the books are right there on the desk. This is called a "hallucination."

The problem is that in a long, detailed answer, the AI might get 90% of the facts right from the books, but slip up on just one or two sentences. Most current tools are like a security guard who only checks the entire report at the end. If the report is mostly good, they might say, "Looks fine!" and miss the tiny lie hidden in the middle.

This paper introduces CORTEX, a new tool designed to be a "microscope" that checks the answer word-by-word (or token-by-token) to spot exactly where the AI is making things up.

Here is how CORTEX works, using three simple analogies:

1. The "With and Without" Test (Comparative Internal Representations)

Imagine you are trying to figure out if a student is actually reading a textbook or just guessing.

The Old Way: You look at the student's answer and try to guess if they read the book.
The CORTEX Way: You ask the student the same question twice.
1. First time: You give them the textbook to read.
2. Second time: You take the textbook away and ask them to answer from memory.

CORTEX compares the "brain activity" (internal thoughts) of the AI during these two scenarios.

If the AI is talking about a fact found in the book, its "brain" changes significantly when the book is present. It's like the student's eyes lighting up because they found the answer in the text.
If the AI is making something up (hallucinating), its "brain" looks almost the same whether the book is there or not, because it's just pulling from its own memory.

By comparing these two states, CORTEX can spot exactly which words changed because of the book (good) and which words stayed the same because they were made up (bad).

2. The "Chain of Thought" Detective (Contextual Residual)

Sometimes, an AI is smart enough to read a fact from the book early in the answer, and then use that fact to build the rest of the sentence later.

The Problem: If you only look at the later sentence, it might look like the AI isn't using the book anymore, because the book isn't mentioned right there. This could trick a detector into thinking the later sentence is a lie, even though it's based on a true fact found earlier.
The CORTEX Solution: CORTEX acts like a detective who follows the trail. It realizes, "Hey, this sentence is relying on a fact the AI just read two sentences ago." It subtracts that "indirect influence" so it doesn't get confused. This prevents the tool from falsely accusing the AI of lying when it's actually just continuing a true story.

3. The "Noise Filter" (Label-Persistence Smoothing)

Imagine you are listening to a radio station that has a lot of static. Sometimes the static makes a single word sound like a lie, even if the whole sentence is true.

The Problem: Raw detection tools can be jittery. They might flag one word as a lie, then the next word as truth, then the next as a lie, creating a messy, scattered pattern. But in reality, if an AI is lying, it usually lies for a whole phrase or sentence, not just one isolated word.
The CORTEX Solution: CORTEX applies a "smoothing" filter. It assumes that if one word is a lie, the words immediately next to it are likely part of the same lie. It connects the dots, turning a scattered mess of "maybe-lies" into a clear, solid block of "definite-lies." This makes the final result much easier for a human to read and understand.

The Results

The authors tested CORTEX on two different sets of questions and three different AI models. They found that:

CORTEX is much better at finding the exact words that are made up compared to other methods.
It works even when the AI is an "API" model (a black box you can't see inside) by using a different, open AI model to do the detective work behind the scenes.
Every part of the system (the comparison, the chain-of-thought check, and the smoothing) helped make the detection more accurate.

In short, CORTEX is a tool that helps us trust AI answers more by pinpointing exactly where the AI is making things up, rather than just guessing if the whole answer is good or bad.

Technical Summary: CORTEX

Problem Statement

Retrieval-Augmented Generation (RAG) is widely used to ground Large Language Model (LLM) outputs in external references, yet models frequently generate hallucinated content even when references are provided. Existing hallucination detection methods face two primary limitations:

Granularity: Most approaches operate at the "answer level," assessing the entire response as a single unit. This is insufficient for long-form RAG outputs where faithful and hallucinated content often coexist.
Methodology: Many existing methods rely on self-consistency (requiring multiple generations) or prompt-based fact-checking (requiring additional inference and prompt engineering), which incur high computational costs. Furthermore, many representation-based detectors analyze a single internal state in isolation rather than comparing how the model's state changes when references are present versus absent.

The paper proposes a need for a token-level detection method that can localize ungrounded content within mixed outputs and operate as a practical, post-hoc framework for both open-weight and closed-weight (API-based) LLMs.

Methodology: CORTEX

CORTEX (Comparative Observed Reference-based Token-level EXpressions) is a post-hoc hallucination detection framework designed specifically for RAG. It utilizes an open-weight LLM ( $M_{open}$ ) as an analysis model to inspect the internal representations of a generated answer, which may have been produced by a closed-weight LLM ( $M_{close}$ ).

The framework operates through three core components:

1. Reference-Induced Delta Representation

CORTEX constructs a paired counterfactual view of the same answer under two conditions:

Reference-conditioned input: $x_{ref} = \text{question} \parallel \text{references} \parallel \text{answer}$
No-reference input: $x_{no-ref} = \text{question} \parallel \text{answer}$

By processing both inputs through $M_{open}$ , the method extracts token-level internal representations ( $h_i$ ) for each answer token $t_i$ . It computes a delta representation ( $\Delta h_i$ ) representing the change in the token's state when references are provided:
$\Delta h_i = h_{i}^{ref} - h_{i}^{no-ref}$
The intuition is that faithful tokens exhibit coherent representational shifts when grounded in references, whereas hallucinated tokens show weaker or incoherent changes.

2. Attention-Based Contextual Residual

In long-form generation, reference influence can propagate indirectly. A token $t_i$ might be grounded not by a direct link to the reference, but by preceding tokens ( $t_{<i}$ ) that were grounded. Relying solely on $\Delta h_i$ might misclassify these indirectly grounded tokens as hallucinations.

To address this, CORTEX calculates a contextual residual ( $c_i$ ). It estimates the reference influence mediated through preceding tokens using attention weights ( $\alpha_{ij}^{ref}$ ) from the reference-conditioned input:
$\bar{\Delta h}_i = \sum_{j<i} \alpha_{ij}^{ref} \Delta h_j$
$c_i = \Delta h_i - \bar{\Delta h}_i$
The residual $c_i$ isolates the token-specific change that remains after accounting for influence propagated from the context, reducing false positives for tokens indirectly grounded in the reference.

3. Token-Level Classification and Smoothing

The features $[\Delta h_i; c_i]$ are fed into a Multilayer Perceptron (MLP) classifier to produce a raw hallucination score $s_i$ .

To align with human annotation patterns (where hallucinations typically appear as contiguous spans rather than isolated tokens), CORTEX applies Label-Persistence Smoothing as a post-processing step. This models the tendency of hallucination labels to persist across neighboring tokens using a pairwise persistence term ( $\rho$ ) and a token-level confidence derived from $s_i$ . The final smoothed score is computed via the forward-backward algorithm, reducing local noise while preserving token-level granularity.

Key Contributions

First Token-Level RAG Detector: To the authors' knowledge, CORTEX is the first token-level hallucination detection method specifically designed for RAG, enabling fine-grained localization of ungrounded content.
Practical Post-Hoc Framework: The method does not require modifying the generator's parameters. It can analyze outputs from arbitrary LLMs, including API-based closed-weight models, by using a separate open-weight model for feature extraction.
Comparative Representation Approach: Unlike prior work that analyzes single internal states, CORTEX leverages the contrast between reference-conditioned and no-reference internal representations to isolate reference-induced signals.
Contextual and Span-Aware Design: The introduction of the contextual residual addresses indirect grounding, while label-persistence smoothing aligns predictions with span-based annotation structures.

Experimental Results

The authors evaluated CORTEX on two benchmarks (RAGTruth and HalluRAG) using three LLMs (Llama-3.1-8B, Qwen3-8B, and Mistral-7B).

Token-Level Performance: CORTEX substantially outperformed five baselines (including NLL, SAPLMA, LLM-Check, ICR Probe, and RAGLens) in terms of Average Precision (AP) and AUROC. On RAGTruth, CORTEX achieved an AP of 0.5940 (Qwen) compared to the next best (RAGLens) at 0.4575.
Answer-Level Performance: Although trained for token-level detection, CORTEX achieved competitive answer-level performance (using the maximum token score as the answer score), demonstrating that token-level signals are effective for global reliability assessment.
Ablation Studies: Removing the contextual residual ( $c$ ) or the smoothing step resulted in performance drops, confirming that both components are necessary for robust detection. The smoothing step was shown to be broadly beneficial even when applied to baseline methods.

Significance and Claims

The paper claims that CORTEX demonstrates the effectiveness of comparing internal representations under different conditioning (with vs. without references) as a signal for hallucination detection.

Practicality: By decoupling the generator from the analyzer, CORTEX offers a viable solution for reliability assessment in deployed systems where the generator is a black-box API.
Granularity: It addresses the limitation of answer-level detection by providing a mechanism to localize hallucinations to specific tokens, which can then be aggregated to any required granularity (words, sentences, paragraphs).
Scope: The authors explicitly note that CORTEX is designed for reference-grounded generation. It detects content unsupported by provided references, which is crucial for controlled applications (e.g., enterprise support) but may flag useful parametric knowledge in open-ended scenarios. The paper does not claim to detect general factual incorrectness independent of the provided context.

The authors conclude that these comparative internal-representation signals could also serve as a basis for future hallucination mitigation strategies, such as reward modeling for tuning generators.