💬 NLP

Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs

This paper introduces the ASL Minimal Translation Pairs (ASL-MTP) benchmark to evaluate how well sign language models capture linguistic phenomena, revealing through a case study that state-of-the-art translation models rely heavily on manual cues while often failing to utilize crucial non-manual information.

Original authors: Serpil Karabüklü, Kanishka Misra, Shester Gueuwou, Diane Brentari, Greg Shakhnarovich, Karen Livescu

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Serpil Karabüklü, Kanishka Misra, Shester Gueuwou, Diane Brentari, Greg Shakhnarovich, Karen Livescu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to understand American Sign Language (ASL). You've given it a massive library of videos showing people signing, and it has learned to translate those videos into English text. It seems smart, right? But here's the problem: Does it actually understand the grammar and nuance of sign language, or is it just guessing based on the most obvious parts?

This paper is like a specialized "eye exam" for that robot. The researchers built a new test called ASL-MTP (American Sign Language Minimal Translation Pairs) to see exactly what the robot is paying attention to.

Here is how the study works, explained through simple analogies:

1. The "Minimal Pair" Test (The "Spot the Difference" Game)

In linguistics, a "minimal pair" is like two sentences that are identical except for one tiny word that changes the whole meaning.

Sentence A: "The movie starts at 7."
Sentence B: "The movie starts at 8."

The researchers created a dataset of 1,275 such pairs. They took a video of someone signing and created two English translations:

The Matched Version: The correct translation of the video.
The Mismatched Version: A translation that is grammatically perfect but wrong for the video (e.g., changing "7" to "8" or turning a question into a statement).

The Test: They asked the AI, "Which of these two sentences fits the video better?" If the AI is truly smart, it should be very confident about the correct one and very confused (surprised) by the wrong one.

2. The "Blindfold" Experiment (Cue Ablation)

ASL isn't just about hand movements. It's a full-body language that uses:

Manual Cues: Hands and fingers.
Non-Manual Cues: Facial expressions (eyebrows, mouth), head tilts, and body posture.

To see what the AI relies on, the researchers played a game of "hide and seek" with the video data. They created different versions of the input where they digitally "blacked out" or removed specific parts of the video:

No Hands: The AI can only see the face and body.
No Face: The AI can only see the hands.
No Eyes/Eyebrows: The AI can see the mouth but not the eyebrows.

They then ran the "Spot the Difference" test again to see if the AI's performance dropped when a specific "blindfold" was applied.

3. The Findings: What the Robot Actually Learned

The Good News:
The AI is generally good at the basics. When it can see everything, it performs better than random guessing on almost all the tests. It is particularly good at reading hands. If you cover up the hands, the AI gets confused about numbers, fingerspelling (spelling words with fingers), and specific hand-shapes. This makes sense because hands carry the "meat" of the message in sign language.

The Bad News (The "Face Blindness"):
The AI is surprisingly bad at reading faces and body language, even though it was trained to see them.

The Eyebrow Problem: In ASL, raising your eyebrows can turn a statement into a question (e.g., "You are ready" vs. "Are you ready?"). When the researchers covered up the AI's view of the eyebrows, the AI didn't seem to care. It still thought the statement was a statement.
The "Declarative Bias": The AI has a strong habit of assuming everything is a statement. Even when the video clearly shows a question (via facial cues), the AI often ignores it and translates it as a statement. It's like a student who refuses to raise their hand to ask a question, so the teacher assumes they are just making a comment.

The "Body Language" Gap:
The AI also struggled with cues that require a mix of hands and body movement (like conditional statements starting with "If..."). When the body or face was hidden, the AI's understanding of these complex sentences fell apart, suggesting it wasn't really "watching" those parts of the video effectively.

4. Why Standard Tests Failed

The researchers also tried using standard translation scores (like BLEURT), which are like a "grade" for how close the translation is to the original text.

The Metaphor: Imagine a student who writes a sentence that is grammatically perfect but answers the wrong question. A standard grader might give them an 'A' because the grammar is perfect.
The Reality: The standard scores couldn't tell the difference between a model that truly understood the sign language grammar and one that was just guessing. The "Minimal Pair" test was the only way to catch the AI's specific blind spots.

Summary

The paper concludes that while current AI models for sign language are impressive, they are over-reliant on hand movements and under-utilize facial expressions and body language. They are like a person who can read a book perfectly but misses the tone of voice, the sarcasm, and the questions being asked.

The researchers hope their new test (ASL-MTP) will help future developers build AI that doesn't just "see" the hands, but truly "sees" the whole person signing.

1. Problem Statement

Sign language (SL) translation models have recently improved in performance on tasks like isolated sign recognition and continuous translation. However, there is a critical gap in understanding how well these models capture specific linguistic phenomena inherent to sign languages.

Unlike spoken languages, sign languages rely on multiple articulatory channels:

Manual: Handshapes, movements, and locations.
Non-manual: Facial expressions (eyebrows, mouth), head movements, and body posture.

Current evaluation metrics (e.g., BLEU, BLEURT) are global similarity measures that fail to detect whether a model is actually sensitive to specific linguistic cues (e.g., distinguishing a question from a statement based solely on an eyebrow raise). It remains unclear if state-of-the-art (SOTA) models utilize non-manual cues effectively or if they rely disproportionately on manual cues, potentially missing crucial grammatical information encoded in the face and body.

2. Methodology

A. Dataset Construction: ASL-MTP

The authors introduce ASL Minimal Translation Pairs (ASL-MTP), a benchmark dataset designed to evaluate fine-grained linguistic capabilities in ASL-to-English translation models.

Source: Derived from the ASLLRP dataset (2,048 linguistically annotated ASL utterances).
Structure: Contains 1,275 video-sentence pairs. For each ASL video, the dataset provides:
1. A Matched translation (grammatically correct and semantically accurate to the video).
2. A Mismatched translation (grammatically correct English, but semantically incorrect regarding the specific linguistic phenomenon in the video).
Phenomena Covered: The dataset targets 9 specific linguistic phenomena, categorized by their primary cues:
- Manual-only: Numbers, Fingerspelling, Classifiers.
- Manual + Non-manual: Negation, Wh-Questions, Conditionals.
- Non-manual dominant: Polar Questions (distinguished from declaratives solely by eyebrow raises).

B. Evaluation Metric: Surprisal Analysis

Instead of generating full translations and comparing them to references, the authors use a surprisal-based analysis:

Input: The model is conditioned on the ASL video ( $F$ ).
Measurement: The model calculates the conditional log-probability (surprisal) for both the matched sentence ( $a$ ) and the mismatched sentence ( $u$ ).
$\Delta \text{Surprisal} = S(u) - S(a)$
Accuracy: A model is considered to have mastered a phenomenon if $\Delta \text{Surprisal} > 0$ (i.e., it finds the mismatched sentence significantly more surprising/unlikely than the matched one). Accuracy is the proportion of pairs where this condition holds. Chance performance is 50%.

C. Case Study Model: SHUBERT+ByT5

The authors analyze SHUBERT+ByT5, a SOTA open-source model.

Architecture: SHUBERT (a self-supervised video encoder trained on 1,000 hours of ASL) combined with ByT5 (a text translation model).
Input Channels: The encoder processes four distinct channels: Face (eyes/mouth), Left Hand, Right Hand, and Body Pose.
Cue Ablation Strategy: To test cue sensitivity, the authors perform two types of ablation:
1. Inference-time Ablation: Masking specific regions (hands, face, body) in the input video frames during testing.
2. Controlled Rearing (Training-time Ablation): Training new variants of the model where specific channels are permanently masked during the training phase to rule out "out-of-distribution" effects.

3. Key Contributions

ASL-MTP Dataset: The first minimal-pair dataset specifically designed for sign language translation, focusing on phenomena-specific sensitivity rather than general translation quality.
Targeted Analysis Framework: A methodology to quantify model sensitivity to specific linguistic cues (manual vs. non-manual) using surprisal differences, which is more diagnostic than standard BLEURT scores.
Empirical Evidence of Bias: A comprehensive analysis revealing that even SOTA models exhibit a strong bias toward manual cues and often fail to utilize critical non-manual cues, despite being trained on multi-channel data.

4. Key Results

Performance on Phenomena (All Cues)

The model performs above chance (50%) on 8 out of 11 subsets (including Numbers, Fingerspelling, and Wh-Questions).
Failure Case: The model performs significantly below chance on Polar Questions vs. Declaratives. It exhibits a strong bias toward generating declarative sentences, failing to recognize that an eyebrow raise (the only cue in these specific stimuli) indicates a question.

Cue Ablation Findings

Heavy Reliance on Hands: When hand channels are masked (No Hands condition), performance drops drastically for manual-heavy phenomena (Numbers, Fingerspelling, Classifiers) and often falls to chance levels.
Insensitivity to Non-Manual Cues:
- Masking the face or body (No Face, No Body) has little to no negative impact on phenomena that theoretically rely on them (e.g., Polar Questions, Conditionals, Negation).
- The model's accuracy on these phenomena remains similar to the "All Cues" baseline even when the critical non-manual cues are removed.
- This suggests the model is not learning to extract grammatical information from facial/body movements, even when those cues are present during training.

Comparison with Standard Metrics

BLEURT vs. Surprisal: Standard translation metrics (BLEURT) failed to detect the model's insensitivity to non-manual cues. BLEURT scores remained relatively stable across ablations, whereas the surprisal-based accuracy showed clear degradation or lack of sensitivity. This confirms that global metrics mask specific linguistic failures.

Controlled Rearing

Training models with ablated channels (e.g., no face data during training) did not significantly improve performance on non-manual phenomena compared to inference-time ablation. This confirms that the failure to use non-manual cues is a fundamental limitation of the current architecture/training objective, not just a distribution mismatch issue.

5. Significance

Diagnostic Power: The paper demonstrates that standard translation metrics are insufficient for evaluating sign language models. Minimal pair analysis is essential for uncovering whether models truly understand the linguistic structure of sign languages.
Model Limitations: The study reveals a critical flaw in current SOTA models: they treat sign language primarily as a "hand gesture" problem, largely ignoring the grammatical weight of non-manual markers (facial expressions, body posture).
Future Directions: The authors call for future models to be explicitly designed to integrate non-manual cues more effectively. ASL-MTP provides a standardized benchmark to track progress in this area and encourages the development of similar benchmarks for other sign languages.

In summary, while current models are improving at translating sign language, they are doing so by relying heavily on manual cues and often failing to capture the complex, multi-channel grammatical rules that define sign languages, particularly those encoded in the face and body.