💬 NLP

Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation

This paper proposes a novel token-level visual reliability measure that combines feature sensitivity and counterfactual signals to effectively detect and diagnose hallucinations in sign language translation by quantifying the extent to which models rely on visual evidence rather than language priors.

Original authors: Yasser Hamidullah, Koel Dutta Chowdhury, Yusser Al Ghussin, Shakib Yazdani, Cennet Oguz, Josef van Genabith, Cristina España-Bonet

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yasser Hamidullah, Koel Dutta Chowdhury, Yusser Al Ghussin, Shakib Yazdani, Cennet Oguz, Josef van Genabith, Cristina España-Bonet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Blind" Translator

Imagine a translator who is trying to translate a video of a person signing (using hand gestures) into spoken text.

The Goal: The translator should look at the hands and say exactly what they are saying.
The Problem: Sometimes, the translator gets lazy or overconfident. Instead of looking at the hands, they just guess what the person probably said based on how sentences usually go. They might produce a sentence that sounds perfect and grammatically correct, but it has nothing to do with what the signer actually did. In AI terms, this is called a hallucination.

This paper argues that the newer, "smarter" translators (which skip an intermediate step called "glosses") are actually worse at this. They are so good at guessing based on language patterns that they often stop looking at the video entirely.

The Core Idea: "Grounding" vs. "Guessing"

The authors introduce a new way to check if the AI is actually looking at the video or just making things up. They call this Reliability.

Think of it like a detective checking an alibi:

Grounding (Looking at the evidence): The AI says, "I am choosing this word because I see the signer's hand moving in a specific way."
Guessing (Relying on memory): The AI says, "I am choosing this word because, in English, people usually say 'hello' after 'good morning'."

The paper's main job is to build a "lie detector" that can tell the difference between these two behaviors.

How the "Lie Detector" Works

The authors created a test to see how much the AI relies on the video. They run the translation three times in parallel:

The Real Deal: The AI sees the actual video.
The Blank Screen: The AI sees the video, but the screen is black (no visual data).
The Wrong Video: The AI sees a completely different video (like a cat instead of a signer).

The Test:

If the AI changes its answer significantly when the video is removed or swapped, it means it was Grounded (it was actually looking at the video).
If the AI keeps saying the exact same thing even when the video is gone or wrong, it means it was Guessing (it was just relying on its internal language habits).

They combine these tests into a single score called Reliability. A high score means the AI is paying attention to the video. A low score means it's hallucinating.

The "Gloss-Free" Trap

The paper compares two types of translators:

Gloss-Based (The Old Way): These models use a middleman. They translate the video into "glosses" (simple labels for hand movements) first, then into text. It's like having a strict teacher who forces the student to point to the object before naming it. This keeps them honest.
Gloss-Free (The New Way): These models jump straight from video to text. They are like a student who is told, "Just tell me what you see," without the strict labels.

The Finding: The "Gloss-Free" models are much more prone to hallucinations. Because they aren't forced to pause and label the movements, they tend to skip looking at the video and just "fill in the blanks" with what sounds good. The paper proves that these models are "guessing" far more often than the older models.

Why This Matters

Usually, when we check if an AI is lying, we look at how "confident" it sounds. If it says something with high confidence, we assume it's true.

The Paper's Twist: In Sign Language Translation, confidence is a trap. A model can be 100% confident that it's raining, even if the video shows a sunny day, because it's just guessing based on the weather forecast it read earlier.

The authors show that their new Reliability Score is much better at catching these lies than just checking confidence. It works by asking: "Did you actually look at the video to decide this, or did you just guess?"

Summary of Results

It Works: The new "Reliability" score successfully predicts when the AI is hallucinating.
It's Universal: It works on different datasets and different types of AI models.
It Explains the "Why": It proves that the newer, gloss-free models hallucinate more because they stop using the visual information and start relying too much on their language training.
It's a Safety Tool: By combining this visual check with standard text checks, we can get a much clearer picture of when an AI is making things up.

In short, the paper teaches us that for sign language translation, you can't just trust the AI's confidence; you have to check if it's actually watching the video.

Technical Summary: Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation

1. Problem Statement

Sign Language Translation (SLT) is a unique video-to-text task where the visual modality constitutes the source language itself, rather than an auxiliary context. A critical failure mode in modern SLT systems, particularly those utilizing gloss-free architectures and Large Language Model (LLM) backbones, is hallucination. In this context, hallucination is defined as the generation of content tokens that are not supported by the signed video evidence.

While recent gloss-free models have solved scalability issues by removing the need for costly intermediate gloss annotations, they introduce a new vulnerability: the decoder tends to rely heavily on language priors rather than visual input when visual representations are ambiguous or of low quality. This results in fluent but factually incorrect translations. Existing hallucination detection methods, which rely on text-intrinsic signals (e.g., confidence, entropy, perplexity), fail to distinguish between a token that is confidently predicted based on linguistic patterns versus one that is confidently predicted based on actual visual evidence.

2. Methodology

The authors propose a novel reliability measure based on grounding usage, which quantifies the extent to which generated tokens rely on video evidence versus language priors. The method operates at the token level and aggregates to the sentence level using counterfactual analysis.

2.1 Core Intuition

At each decoding step, a model either grounds the next token in the video signal or guesses it from language priors. The proposed method measures this reliance by comparing the model's behavior under the true video input against counterfactual scenarios where the visual input is altered.

2.2 Signal Components

The reliability score is derived from two classes of evidence:

Feature-Based Sensitivity (Internal Reliance):
- Hidden State Sensitivity: Measures the directional change (normalized angular distance) in decoder hidden states when the video input is removed (masked) versus when it is present. A larger angle indicates stronger reliance on the video.
- Cross-Attention Usage: Aggregates the attention mass directed from decoder tokens to video encoder positions. The difference between attention under clean video and masked video indicates the extent of visual grounding.
Counterfactual Signals (External Evidence):
- The system runs three parallel decoding passes conditioned on the same prefix:
  1. Clean: With the true video ( $x$ ).
  2. No-Video: With the visual encoder disabled ( $\emptyset$ ).
  3. Mismatched: With a shuffled or unrelated video ( $x'$ ).
- Probability Margins: The method computes the log-probability and logit differences between the clean pass and the strongest counterfactual (the maximum probability of the no-video or mismatched video passes).
- Advantages: Calculates absolute probability gains ( $\Delta_{clean}$ and $\Delta_{mis}$ ) to determine if the correct video actively improves the prediction compared to alternatives.

2.3 Fusion and Aggregation

Token-Level: A linear fusion of feature-based signals and counterfactual signals, passed through a sigmoid activation, yields a token-level reliability score ( $r_t$ ).
Sentence-Level: Token scores are aggregated using tail-10 pooling (averaging the lowest 10% of token reliabilities). This focuses on the least reliable parts of the sentence, which are most indicative of hallucination risk.
Integration: The grounding-based reliability is combined with standard text-only baselines (confidence, perplexity, entropy) in a "META" variant to create a comprehensive hallucination risk estimator.

3. Key Contributions

First Investigation of SLT Hallucinations: This work is the first to explicitly investigate and characterize hallucinations specifically within the domain of Sign Language Translation.
Grounding Usage Metric: Introduction of a novel reliability measure that quantifies visual grounding, distinguishing between tokens generated via visual evidence and those guessed via language priors.
Counterfactual Framework: Design of a scoring framework using counterfactual perturbations (masking and mismatching video) to capture both internal model state changes and external probability shifts.
Empirical Validation: Demonstration that grounding-based signals outperform text-only baselines in detecting hallucinations and provide complementary information when combined with linguistic uncertainty measures.

4. Experimental Results

The method was evaluated on two benchmarks: PHOENIX-2014T (German Sign Language) and CSL-Daily (Chinese Sign Language), using both gloss-based (TwoStream-SLT) and gloss-free (SpaMo) models.

Correlation with Hallucination: There is a strong negative correlation between the proposed reliability score and the CHAIR (Caption Hallucination Assessment with Image Relevance) metric. As grounding usage increases, hallucination rates decrease.
Detection Performance: The grounding-based reliability significantly outperforms text-only baselines (confidence, perplexity, entropy) in binary hallucination detection.
- On CSL-Daily (Gloss-Free), the grounding method achieved an AUC of 0.951 and Accuracy of 0.970, compared to ~0.953 AUC and 0.868 Accuracy for the best text-only baseline (Confidence).
- The combined META variant achieved the highest performance, reaching an AUC of 0.963 and Accuracy of 0.883 on CSL-Daily (Gloss-Free).
Regression Performance: The reliability score effectively predicts the continuous CHAIR rate, achieving a Spearman correlation of $\rho = -0.680$ on CSL-Daily (Gloss-Free).
Generalization: The reliability weights trained on one dataset/model family transfer robustly to others, though performance degrades slightly when transferring from gloss-based to gloss-free models due to differences in tokenization granularity and hallucination prevalence.
Stress Testing: Under visual degradations (Gaussian noise, frame dropping), the reliability score decreases, confirming its sensitivity to visual information quality.
Gloss-Free vs. Gloss-Based: Analysis reveals that gloss-free models hallucinate more frequently because they systematically under-utilize visual information, defaulting to linguistic priors. Token-level distributions show gloss-free models have lower log-margins and smaller hidden-state sensitivity angles compared to gloss-based models.

5. Significance and Claims

The paper claims that hallucination detection in multimodal generation cannot be fully explained by text-intrinsic signals alone. In SLT, where the visual input is the source language, distinguishing between "guessing" (relying on language priors) and "grounding" (relying on visual evidence) is critical.

The authors establish that:

Reliability is a Practical Tool: The proposed measure provides a reusable, interpretable tool for diagnosing hallucinations in SLT without requiring reference translations.
Complementarity: Visual grounding signals provide complementary information to text-based uncertainty measures. Combining them yields the most robust detection framework.
Root Cause of Gloss-Free Errors: The systematic over-reliance on language priors in gloss-free models is the primary driver of their higher hallucination rates. The lack of intermediate gloss supervision allows the decoder to drift from visual evidence.
Foundational Work: This research lays the groundwork for more robust hallucination detection in multimodal generation by explicitly disentangling the contribution of visual grounding from linguistic fluency.

The paper concludes that a robust multimodal model must strike a balance between guessing (for fluency) and grounding (for factual correctness), and that explicitly measuring this balance is essential for reliable SLT systems.