💬 NLP

MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probing

MultiHaluDet is a novel, language-agnostic three-stage framework that detects multilingual hallucinations in frozen Large Language Models by probing full hidden state trajectories with a hybrid attention architecture, achieving state-of-the-art performance and robust cross-lingual generalization across high, medium, and low-resource languages without requiring language-specific fine-tuning.

Original authors: Riasad Alvi, Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Riasad Alvi, Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot (a Large Language Model, or LLM) that loves to tell stories and answer questions. Sometimes, this robot gets confident but makes things up completely. These made-up facts are called hallucinations.

The paper introduces a new tool called MULTIHALUDET (Multilingual Hallucination Detection). Think of it not as a fact-checker that Googles the answer, but as a lie detector that listens to the robot's heartbeat.

Here is how it works, broken down into simple concepts:

1. The Problem: Why Old Methods Fail

Previous ways of catching lies were like asking the robot, "Are you sure?"

The Confidence Trap: If the robot says, "I am 100% sure," old methods thought it was telling the truth. But the paper shows the robot can be confidently wrong. It's like a smooth-talking liar who is very convincing.
The Single-Check Trap: Other methods looked at just one part of the robot's brain (one layer) or just the final word it spoke. The paper argues that lies often hide in the journey of the thought process, not just the destination.

2. The Solution: Listening to the "Brainwaves"

MULTIHALUDET doesn't ask the robot what it thinks. Instead, it probes the robot's internal "hidden states" while it is thinking.

The Analogy: Imagine the robot is writing a story.
- Old Method: Reads the final sentence to see if it makes sense.
- MULTIHALUDET: Watches the robot's hand as it writes every single letter, feeling the pressure, the speed, and the hesitation at every step. It looks for "tremors" in the writing process that signal a lie, even if the final sentence looks perfect.

3. How It Works (The Four Stages)

The system acts like a four-step detective agency:

The Scan (Feature Extraction): The robot answers a question. The system freezes the robot (doesn't change its brain) and records a "video" of its internal thoughts as it moves from the first layer of its brain to the last.
The Zoom Lens (Multi-Scale Attention): The system doesn't just look at the whole video or just one frame. It uses a "zoom lens" to look at the big picture and the tiny details simultaneously. It looks for sudden shifts or weird patterns in how the robot's thoughts evolve.
The Team Huddle (Ensemble Meta-Learner): Instead of one detective making the call, the system uses a team of different experts (like a tree-based detective, a math-based detective, and a neural network detective). They all vote on whether the robot is lying.
The Safety Net (Out-of-Fold Stacking): To make sure the team doesn't cheat by memorizing the answers, they practice in a way where they never see the test questions during training. This ensures they are actually learning to spot lies, not just memorizing facts.

4. The Multilingual Superpower

Most lie detectors only work well in English. MULTIHALUDET is special because it works in French, Bangla, and Amharic (a language with very few digital resources) without needing to be retrained for each language.

The Result: It works almost as well in French as it does in English. In Bangla and Amharic, it still does a great job, far better than any previous method, even though the robot isn't as "fluent" in those languages. It proves that the "tremors" of a lie look similar in the robot's brain, no matter what language it is speaking.

5. The Results

The paper tested this on two big sets of questions (HaluEval and TriviaQA).

The Score: While the best previous methods got about 95% correct, MULTIHALUDET hit 98.5%.
The Robustness: It worked equally well on two different types of robot brains (Mistral-7B and LLaMA2-7B), proving it's not just lucky with one specific model.

6. The Catch (Limitations)

The paper is honest about what it can't do:

White-Box Only: You need to be able to see inside the robot's brain to use this. You can't use it on closed, secret robots (like GPT-4) where you can't see the internal "heartbeat."
Heavy Lifting: It requires the robot to do a full "forward pass" (think through the whole answer) and record all the data, which uses more computer memory than just asking "Are you sure?"

Summary

MULTIHALUDET is a high-tech stethoscope for AI. Instead of trusting what the AI says, it listens to how the AI thinks. By analyzing the subtle, complex patterns of the robot's internal thoughts across many languages, it can spot when the robot is making things up with incredible accuracy.

Technical Summary: MULTIHALUDET

Problem Statement

Hallucinations in Large Language Models (LLMs) pose a critical barrier to their reliable deployment, particularly in non-English and resource-constrained contexts. Existing detection methods face significant limitations:

Evidence-based methods rely on external retrieval, introducing computational latency and dependency on retrieval quality.
Confidence-based heuristics (e.g., logit probabilities, entropy) often fail to capture deep factual inconsistencies, as hallucinations can manifest as semantic-level confabulations even when the model exhibits high output confidence.
Internal state probing methods often focus on single tokens or specific layers, struggling when non-factual tokens are distributed across sequences or when models vary in depth. Furthermore, these methods frequently lack robustness across typologically diverse languages without language-specific fine-tuning.

Methodology: MULTIHALUDET

The authors propose MULTIHALUDET, a four-stage, supervised framework designed to detect multilingual hallucinations by probing the full hidden state trajectories of frozen LLMs. The framework operates without requiring language-specific fine-tuning.

1. LLM-Based Feature Extraction

The process begins by passing structured prompts through a frozen, quantized LLM to extract hidden state tensors $\{H^{(l)}\}_{l=0}^L$ .

Dynamic Layer Sampling: To ensure compatibility across models of varying depths ( $L$ ), a dynamic sampling strategy maps layers to a fixed target count ( $K=32$ ). This involves identity mapping for shallow models, repetition for padding, and uniform interpolation for deep models.
Sequential Per-Layer Features: For each sampled layer, the framework extracts compact descriptors from both the last-token representation and the sequence mean. These descriptors include distributional statistics (e.g., $\ell_2$ norm, mean, standard deviation, sparsity, kurtosis, and Median Absolute Deviation) to capture the evolution of internal dynamics.
Anchor-Based Depth Probing: To enable cross-architecture comparison, features are extracted from four "anchor" layers corresponding to proportional depth fractions (early, middle, late stages), ensuring consistent semantic representation regardless of total model depth.
Global Feature Vector: A global vector aggregates next-token probabilities, logit statistics, layer-wise norm trajectories, and cross-feature interaction terms between output confidence and internal geometry.

2. MultiHaluDet Architecture

The extracted features are processed through a deep learning architecture:

Multi-Scale Attention: The sequential input is projected and processed at multiple temporal resolutions. A learned position-wise gate combines contributions from different scales, enriching fine-grained layer information with multi-scale context.
Layer-Weighted Transformer Encoder: A learnable importance vector modulates the fused sequence before it is encoded by a stack of Pre-LN Transformer layers. This allows the model to attend to specific LLM depths and capture long-range inter-layer dependencies.
Self-Attention Pooling: Instead of mean pooling, a learned attention mechanism assigns relevance scores to each position, enabling input-adaptive depth selection to focus on the most informative layers.
Gated Fusion: The sequential representation is concatenated with the global feature vector. A sigmoid gate performs element-wise re-weighting to suppress uninformative dimensions before passing the result to a classifier.

3. Out-of-Fold (OOF) Stacking

To prevent data leakage and obtain unbiased deep representations, the framework employs K-fold OOF stacking. A fresh instance of the model is trained on $K-1$ folds to generate gated fusion representations for the held-out fold. These OOF features form a robust training matrix for the meta-learner.

4. Ensemble Meta-Learner

The OOF deep features serve as input to a diverse ensemble of classifiers (Gradient Boosted Trees, Random Forests, SVM, MLP, Logistic Regression). A logistic meta-regressor stacks the probability estimates of these base learners, adapting the ensemble composition to the discriminative strength of each component.

5. Multilingual Adaptation

To evaluate cross-lingual generalization, English datasets (HaluEval and TriviaQA) were translated into French (high-resource), Bangla (medium-resource), and Amharic (low-resource) using Gemini 2.5 Flash. Rigorous human evaluation and back-translation ensured semantic integrity. The frozen LLM processes these inputs natively in their respective token spaces, allowing the detector to probe language-agnostic hallucination signals.

Key Contributions

Framework Design: Introduction of MULTIHALUDET, a four-stage framework combining dynamic feature extraction, multi-scale attention, OOF deep feature generation, and learned ensemble meta-learning to capture both fine-grained and coarse-grained patterns of factual inconsistency.
Performance Gains: Demonstration of state-of-the-art detection performance on HaluEval and TriviaQA benchmarks using Mistral-7B and LLaMA2-7B, achieving up to 98.55% AUROC.
Cross-Lingual Generalization: Comprehensive evaluation showing that the framework maintains strong detection signals across typologically diverse linguistic tiers (French, Bangla, Amharic) without language-specific fine-tuning.

Results

Benchmark Performance: MULTIHALUDET significantly outperforms 13 baselines, including confidence-based methods (P(True), AvgProb), consistency-based methods (INSIDE, Unigram), and other probing approaches (SAPLMA, MIND, Neural ODEs/CDEs/SDEs).
- On HaluEval with LLaMA2-7B, it achieved 98.55% AUROC, compared to the next best baseline (Neural CDEs) at 92.8%.
- On TriviaQA with LLaMA2-7B, it achieved 98.26% AUROC, compared to 83.7% for the next best.
Cross-Architecture Robustness: Unlike baselines that suffer performance drops when switching from Mistral-7B to LLaMA2-7B, MULTIHALUDET maintained near-identical efficacy across architectures.
Cross-Lingual Robustness:
- French: Performance remained high (96.2% on HaluEval), trailing English baselines only marginally.
- Bangla: Showed controlled degradation (89.1% on HaluEval) but significantly outperformed baselines (78.4%).
- Amharic: Despite being a low-resource language where base models struggle with factual recall, MULTIHALUDET achieved 78.5% on HaluEval, maintaining a robust lead over random chance and standard confidence baselines.
Ablation Studies: Removing the OOF Stacking mechanism caused the most severe performance drop (~~10%), followed by the removal of Multi-Scale Attention (~~7%). Replacing trajectory probing with static final-layer representation resulted in a ~5% drop, confirming the necessity of analyzing the full hidden state evolution.

Significance and Limitations

The paper claims that MULTIHALUDET demonstrates that internal state probing can effectively detect hallucinations across diverse languages and model architectures without fine-tuning. It addresses the limitation of current methods that fail to capture semantic-level confabulations in non-English contexts.

However, the authors acknowledge specific limitations:

White-Box Requirement: The framework requires access to internal hidden states and logits, making it inapplicable to proprietary black-box models (e.g., GPT-4, Claude).
Computational Overhead: While it avoids fine-tuning, extracting and processing full-depth hidden states incurs non-trivial memory overhead and requires complete forward passes.
Translation Artifacts: The multilingual evaluation relies on translated datasets. While quality assurance was rigorous, native, naturally occurring prompts in low-resource languages might reveal nuances not fully captured by translated benchmarks.