💬 NLP

HALT: Hallucination Assessment via Log-probs as Time series

The paper introduces HALT, a lightweight and efficient hallucination detector that analyzes token log-probabilities as time series to achieve superior performance and speed compared to existing methods, alongside HUB, a comprehensive benchmark unifying ten diverse capabilities for evaluating hallucination detection across large language models.

Original authors: Ahmad Shapiro, Karan Taneja, Ashok Goel

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmad Shapiro, Karan Taneja, Ashok Goel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are listening to a storyteller tell a tale. Sometimes, they speak with absolute confidence, their voice steady and clear. Other times, they might stutter, hesitate, or their voice might waver right before they make up a detail that isn't true.

For a long time, researchers trying to catch Large Language Models (LLMs) lying (or "hallucinating") have tried two main approaches:

The "White-Box" approach: Asking the storyteller to show their secret notes and brainwaves. This is great if you own the storyteller, but impossible if you are just using a public API (like a chatbot).
The "Black-Box" approach: Asking the storyteller to repeat the story or asking a second storyteller to fact-check them. This is slow, expensive, and sometimes the second storyteller lies too.

HALT (Hallucination Assessment via Log-probs as Time series) is a new, clever detective that needs neither the secret notes nor a second storyteller. It only listens to the rhythm of the storyteller's confidence.

The Core Idea: Listening to the "Heartbeat" of Confidence

When an AI generates text, it doesn't just pick a word; it calculates how likely every possible word is to come next. These calculations are called log-probabilities. Think of these as the AI's internal "confidence meter" for every single word it says.

The authors of this paper realized that these confidence meters don't just sit there; they move like a time series (a heartbeat or a stock market chart).

When an AI is telling the truth, its confidence meter usually follows a smooth, steady rhythm.
When it starts to hallucinate (make something up), that rhythm gets weird. It might spike, drop suddenly, or wobble in a specific pattern, even if the AI sounds very confident.

HALT is a tiny, lightweight computer program (a "GRU" model) trained to watch this confidence rhythm. It doesn't read the words the AI says; it only watches the graph of the AI's confidence as it speaks.

The Detective's Toolkit: HALT

HALT is like a specialized stethoscope. It looks at the top 20 most likely words the AI is considering at every step. It measures:

How shaky the choice is: Is the AI torn between two words? (High uncertainty).
How sudden the change is: Did the AI suddenly jump from being 99% sure to 50% sure?
The pattern over time: Does the confidence curve look like a smooth hill or a jagged mountain range?

By feeding this "confidence heartbeat" into its brain, HALT learns to spot the specific "arrhythmia" that happens when an AI starts lying.

The New Stadium: HUB

To test if HALT actually works, the authors built a massive new testing ground called HUB (Hallucination detection Unified Benchmark).

Imagine previous tests were like playing only in a small, quiet library (focusing only on simple facts or chat). HUB is a giant, chaotic stadium with 10 different sports:

Reasoning Sports: Math, coding, logic puzzles, and algorithms.
General Sports: Chatting, summarizing news, and answering trivia.

The authors realized that "hallucination" isn't just about making up facts (like saying the moon is made of cheese). It's also about logical hallucinations—where an AI gets the facts right but messes up the math or the logic steps to get there. HUB tests all of these.

The Results: Small but Mighty

The paper compares HALT to other detectors:

Lettuce: A very large, heavy detector (like a giant tank) that reads the actual text.
White-box methods: Detectors that need to see the AI's internal brain (which you can't do with most commercial AI).

The Surprise: HALT is 30 times smaller than the heavy tank (Lettuce) and 60 times faster. Yet, it wins more often!

It beats the heavy tank in 7 out of 10 sports.
It works just as well as the giant tank but doesn't need to read the text or see the AI's brain. It just listens to the confidence rhythm.

Important Limitations (What the Paper Says)

It's Model-Specific: HALT is like a coach who knows one specific player's heartbeat perfectly. If you train HALT on an AI named "Llama," it becomes an expert at spotting lies in Llama. If you try to use that same HALT on a different AI named "Qwen," it gets confused. The "heartbeat" patterns are different for different models.
It Needs Access to the "Confidence Numbers": You don't need the AI's brain, but you do need the API to give you the top 20 confidence numbers for every word. If an API hides those numbers completely, HALT can't work.
It Detects, It Doesn't Fix: HALT is a smoke alarm. It tells you when the AI is hallucinating, but it doesn't tell you why or how to fix the story.

Summary Analogy

Think of an AI as a magician performing a trick.

Old detectors tried to peek under the magician's hat (White-box) or ask the audience to vote on whether the trick was real (Black-box text analysis).
HALT just watches the magician's hands. It knows that when a magician is about to pull a fake rabbit out of a hat, their hand movements get slightly jerky and unpredictable, even if they are smiling confidently. HALT is trained to spot that specific "jerky hand" pattern in the AI's confidence numbers, allowing it to catch the lie instantly without needing to see the trick or ask anyone else.

Technical Summary: HALT (Hallucination Assessment via Log-probs as Time series)

1. Problem Statement

Large Language Models (LLMs) frequently produce hallucinations—outputs containing verifiably false or unsupported information. This poses a significant barrier to deploying LLMs in safety-critical domains. Existing detection methods face distinct limitations:

White-box approaches require access to internal model states (hidden activations, attention maps), which is impossible for proprietary, closed-source APIs.
Black-box text-based approaches often rely on external retrieval, additional API calls, or auxiliary LLMs to judge consistency. These introduce latency, cost, and the risk that the "judge" model itself hallucinates or introduces domain bias.
Uncertainty-based approaches often rely on aggregate statistics (e.g., mean confidence, overall entropy) which may fail to capture the temporal dynamics of model calibration errors.

The authors posit that while raw token probabilities alone are not reliable indicators of truthfulness due to calibration biases, the temporal evolution of these probabilities contains stable signals of uncertainty. The central research question is: Can hallucinations be detected by modeling only the sequence of token log-probabilities without analyzing the generated text or consulting external references?

2. Methodology

2.1. Core Paradigm: Log-probs as Time Series

HALT treats the sequence of top- $k$ token log-probabilities as a time-series classification problem. Instead of collapsing probabilities into summary statistics, the method preserves the ordered sequence of the model's confidence at each decoding step.

Input Representation:
For a response of length $T$ , the model extracts the top-20 log-probabilities ( $\ell_t \in \mathbb{R}^{20}$ ) at each timestep $t$ . These are concatenated with five engineered features to form an enriched feature vector $\tilde{\ell}_t$ :

Average Log-Probability: A surrogate for distribution sharpness.
Rank Proxy: A bounded rank of the selected token within the top-20, capturing non-greedy or atypical choices.
Overall Entropy ( $H_{overall}$ ): Uncertainty over the selected token and alternatives.
Alternatives-only Entropy ( $H_{alts}$ ): Disagreement among competing candidates.
Temporal Change in Decision Entropy ( $\Delta H_{dec}$ ): Captures sharp transitions in certainty between the selected token and the best alternative.

The final input is a time series $\tilde{\ell}_{1:T} \in \mathbb{R}^{T \times 25}$ .

2.2. Architecture

The model employs a lightweight Bidirectional Gated Recurrent Unit (GRU) with 5 million parameters.

Encoder: A multi-layer bidirectional GRU processes the sequence to capture temporal patterns in both forward and backward directions.
Pooling: A Top- $q$ pooling mechanism (where $q=0.15$ ) aggregates the hidden states. This averages the most salient timesteps (those with the largest hidden-state norms), focusing on moments of sharp confidence shifts rather than diluting signals across all tokens.
Output: A linear layer produces a single logit for binary classification (hallucinated vs. faithful), trained with binary cross-entropy loss.

2.3. Key Hypotheses

The authors propose four hypotheses validated in the study:

Model-Specific Bias: Each LLM has a deterministic calibration bias function governing its top- $k$ probabilities.
Learnability: A GRU can learn to approximate this bias-induced dynamics, associating specific temporal patterns with hallucinations.
Non-Transferability: Detectors trained on one model's log-probs do not transfer reliably to a different model architecture due to differing calibration biases.
Task Generalization: Within a single model, a detector trained on one task family generalizes to others because the underlying calibration bias is consistent across tasks.

3. Benchmark: HUB (Hallucination detection Unified Benchmark)

To evaluate the method, the authors introduce HUB, a comprehensive benchmark spanning 10 LLM capabilities:

Reasoning-oriented: Algorithmic, Commonsense, Mathematical, Symbolic Reasoning, and Code Generation.
General-purpose: Chat, Data-to-Text, Question Answering, Summarization, and World Knowledge.

HUB extends prior datasets (FAVA, RAGTruth, HaluEval) and integrates CriticBench to include logical hallucinations (flawed reasoning steps leading to incorrect or lucky-correct answers) alongside factual hallucinations. The dataset is split into train, validation, and test sets, with training restricted to Chat, Data-to-Text, and QA to rigorously test out-of-distribution generalization on reasoning tasks.

4. Results

4.1. Performance on HUB

HALT significantly outperforms existing baselines:

Efficiency: HALT is 30 $\times$ smaller than the Lettuce detector (a fine-tuned ModernBERT-base encoder) and achieves a 60 $\times$ speedup.
Accuracy: HALT achieves the best Macro-F1 scores on 7 out of 10 clusters.
- HALT-L (trained on LLaMA 3.1-8B) achieves an overall Macro-F1 of 67.01, outperforming Lettuce (64.00) and aggregate statistics baselines (e.g., $H_{overall}$ at 32.86).
- It excels particularly in reasoning tasks (Algorithmic, Commonsense, Symbolic) where sequence-level calibration cues are more predictive than text-level heuristics.

4.2. Transferability and Generalization

Cross-Task: HALT demonstrates strong generalization across different task families within the same model, supporting the hypothesis that calibration bias is task-agnostic.
Cross-Model: As predicted, performance drops significantly when a model trained on LLaMA log-probs is applied to Qwen log-probs (and vice versa), confirming that calibration dynamics are model-specific and not directly transferable across architectures.
Robustness: HALT remains competitive on external benchmarks (FAVA, RAGTruth) despite not being tuned specifically for them.

5. Significance and Claims

The paper claims the following contributions and significance:

New Black-Box Paradigm: HALT introduces a detection method that relies solely on token log-probabilities, avoiding the need for surface-form text analysis, external validators, or internal model access. This makes it uniquely applicable to proprietary APIs.
Temporal Dynamics over Static Metrics: The work demonstrates that treating log-probabilities as a time series captures uncertainty patterns (e.g., abrupt entropy spikes, rank inversions) that aggregate statistics miss.
Unified Framework: By integrating logical reasoning failures into the definition of hallucination, HUB provides a broader view of semantic unfaithfulness, moving beyond simple factual verification.
Practical Efficiency: The method offers a highly efficient alternative to large encoders, enabling real-time hallucination detection with negligible overhead, as it only requires the log-probability stream often already available during generation.

The authors conclude that while HALT is not tied to a single model, it learns transferable uncertainty dynamics that generalize across tasks within a model, establishing a new modality for analyzing model uncertainty without requiring access to weights or external data.