💬 NLP

DualAnchor: Preserving Language Priors and Improving Lexical Fidelity in Gloss-Free Sign Language Translation

The paper proposes DualAnchor, a gloss-free sign language translation framework that combines Token-level Prior Anchoring to preserve language fluency and Optimal Transport Alignment to enhance lexical fidelity, thereby addressing the common issues of language-prior degradation and lexical gaps in existing LLM-based methods.

Original authors: Hongbin Zhang, Junhao Liu, Xuefeng Bai, Youcheng Pan, Yang Xiang, Kehai Chen

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongbin Zhang, Junhao Liu, Xuefeng Bai, Youcheng Pan, Yang Xiang, Kehai Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to translate a silent movie into a spoken story. This is the world of Sign Language Translation (SLT), a field where computers try to turn videos of people using their hands and faces into written or spoken sentences. For a long time, these robots were like clumsy students who memorized specific hand signs (called "glosses") but struggled to understand the flow of a real conversation. Recently, scientists started using Large Language Models (LLMs)—the same super-smart AI brains that write essays and chat with us—to act as the robot's "brain" for the story part. The idea was simple: give the robot a video, let the AI figure out the words, and boom, you have a translation.

But there's a catch. Even though these AI brains are amazing at writing fluent English or German, when you force them to look at a video, they sometimes forget how to speak properly. They might start spitting out gibberish or forgetting grammar rules because they are too focused on the pictures. Also, they might get the general idea right ("a girl is eating") but mess up the specific details ("eating an apple" vs. "eating an orange"). This paper, titled DualAnchor, tackles these two specific problems: making sure the robot doesn't lose its language skills and making sure it gets the tiny details right.

The Two Big Problems: Losing the Plot and Mixing Up the Details

The researchers noticed that when they tried to teach these AI models to understand sign language videos, two weird things happened.

First, they called it "Language-Prior Degradation." Think of the AI's language skills as a well-rehearsed script it learned from reading millions of books. When you show it a video, it's like asking that actor to improvise on stage. Sometimes, in trying to match the video, the actor forgets the script entirely and starts mumbling nonsense or using bad grammar. The AI gets so distracted by the visual signals that it forgets how to speak fluently.

Second, they found a "Lexical Fidelity Gap." Imagine the AI is a detective looking at a crime scene photo. It might correctly guess that a "person" is holding a "fruit." But if the photo shows an orange and the AI writes apple, it has the general idea right but the specific detail wrong. The paper found that even when the AI matched the video and the sentence perfectly on a big scale, it was still swapping out specific words like "refrigerator" for "cabinet" or "apple" for "orange" because it wasn't looking closely enough at the visual evidence for each individual word.

The Solution: DualAnchor

To fix this, the authors built a new training system called DualAnchor. The name comes from the idea of dropping two anchors to keep a ship steady in rough waters. In this case, the "ship" is the AI model, and the "rough waters" are the confusing visual signals of sign language.

Anchor 1: Token-Level Prior Anchoring (TPA)
This anchor is all about keeping the AI from forgetting how to speak. Imagine the AI is a student taking a test. Usually, when it looks at a video, it might guess a word and get it wrong. With TPA, the researchers give the student a "cheat sheet" from a frozen, super-smart teacher (the original AI before it saw any videos). Every time the student is about to guess the next word, the system checks: "Does this guess sound like something the teacher would say?"

If the teacher is very confident (like when the sentence is "Yesterday, I went..."), the system gently nudges the student to follow the teacher's lead to keep the grammar perfect. But if the teacher is unsure (maybe the video is blurry), the system lets the student look at the video more closely to make its own choice. This way, the AI keeps its fluent language skills while still learning from the video.

Anchor 2: Optimal Transport Alignment (OTA)
This anchor fixes the "wrong details" problem. Instead of just matching the whole video to the whole sentence, OTA acts like a super-precise matching game. It tries to pair up specific parts of the video (like a hand shape showing "apple") with specific words in the sentence ("apple").

The clever part is that it knows not everything in the video has a direct word match. Sometimes a hand wave is just a gesture, not a word. So, the system uses a "trash can" (called a dustbin) to throw away the parts of the video that don't match any word, rather than forcing a bad match. It uses a mathematical trick called "Optimal Transport" to find the best possible way to pair the visual clues with the right words, ensuring that if the video shows a refrigerator, the AI writes "refrigerator," not "cabinet."

What They Found

The researchers tested this new DualAnchor system on two big datasets: PHOENIX-2014T (German sign language weather reports) and CSL-Daily (Chinese sign language daily life videos).

The results were promising. On the German dataset, their method achieved a BLEU-4 score of 27.60, which was the highest among all the "gloss-free" methods they compared against. On the Chinese dataset, they hit 24.21, again beating the competition. The paper suggests that these improvements aren't just random luck; the analysis showed that the "Prior Anchoring" (TPA) really did make the sentences flow better and sound more natural, while the "Optimal Transport" (OTA) significantly reduced the number of wrong words, like swapping fruits or colors.

They also checked if this worked with different types of AI brains (backbones) and found that DualAnchor improved performance across the board, suggesting that this two-anchor approach is a solid way to teach robots to translate sign language without losing their mind or their vocabulary.

In short, DualAnchor teaches the AI to listen to its inner language expert to stay fluent, while simultaneously acting like a sharp-eyed detective to make sure every single word matches the video perfectly. It's a balance between keeping the story smooth and getting the details right.

Technical Summary: DualAnchor for Gloss-Free Sign Language Translation

Problem Statement
Recent advancements in Large Language Models (LLMs) have shifted Sign Language Translation (SLT) toward gloss-free frameworks where pretrained LLMs serve as textual backbones. However, the authors identify two critical, complementary issues in current LLM-based SLT methods that degrade translation quality:

Language-Prior Degradation: Despite the strong language modeling capabilities of LLM backbones, the multimodal adaptation process often undermines the pretrained language prior. The visual adaptation shifts the decoder's next-token distribution away from the linguistic structure acquired during text pretraining, resulting in disfluent and ungrammatical outputs.
Lexical Fidelity Gap: Existing methods typically align videos and text at the sentence level using contrastive objectives. While this achieves strong global semantic alignment, it fails to ensure accurate lexical details. Models can produce globally coherent sentences while mistranslating specific actions, entities, or attributes (e.g., predicting "cabinet" and "orange" instead of "refrigerator" and "apple"), creating a gap between global alignment and fine-grained visual grounding.

Methodology: DualAnchor
To address these issues, the authors propose DualAnchor, a gloss-free LLM-based SLT training framework that couples two complementary training objectives (anchors) to preserve linguistic fluency and improve visual grounding. Both anchors operate only during training and are removed during inference.

Token-Level Prior Anchoring (TPA):
- Goal: Preserve the LLM's pretrained language prior to ensure fluency.
- Mechanism: TPA regularizes the multimodal decoder's next-token distribution at each decoding step. It compares the student model's distribution ( $p_s$ ) against a frozen LLM prior ( $p_p$ ) conditioned on the same autoregressive prefix (the generated text so far).
- Confidence Adaptation: The method employs a confidence gate based on token entropy. If the frozen prior is confident (low entropy), the constraint is strong; if the prior is uncertain (high entropy), the student is allowed more freedom to follow visual evidence.
- Loss: It minimizes the reverse Kullback-Leibler (KL) divergence between the student and the prior, scaled by the confidence gate. This prevents the model from assigning probability mass to tokens unsupported by the pretrained language structure.
Optimal Transport Alignment (OTA):
- Goal: Improve lexical fidelity by aligning visual tokens with textual content tokens.
- Mechanism: OTA formulates visual-textual matching as an entropy-regularized partial optimal transport problem. It matches visual tokens ( $Z_v$ ) with content tokens ( $Z_c$ ) (excluding padding and function words) using a cosine cost matrix.
- Partial Matching: Unlike standard alignment, OTA introduces a "dustbin" (sink) to absorb unmatched mass. This handles the granularity mismatch between continuous visual signals and discrete text tokens, allowing the model to ignore unreliable correspondences rather than forcing a match.
- Loss: The model minimizes the average transport cost of the genuine matches, solved efficiently via Sinkhorn scaling.

Key Contributions

Diagnosis of SLT Issues: The paper identifies and empirically validates that multimodal adaptation causes a fluency gap (increased perplexity relative to references) and that high global video-text alignment does not guarantee fine-grained lexical accuracy.
DualAnchor Framework: The introduction of a training framework that simultaneously anchors linguistic form (via TPA) and grounds lexical choices (via OTA).
State-of-the-Art Performance: Achieving the best BLEU-4 scores among compared gloss-free methods on both PHOENIX-2014T (German) and CSL-Daily (Chinese) benchmarks, along with the highest arithmetic mean across ten reported metrics.

Experimental Results

Datasets: Evaluated on PHOENIX-2014T and CSL-Daily.
Performance: DualAnchor outperforms eight existing gloss-free baselines (including SignLLM, BeyondGloss, and SpaMo). On PHOENIX-2014T, BLEU-4 improved from 24.32 (SpaMo) to 27.60. On CSL-Daily, it improved from 20.55 to 24.21.
Ablation and Analysis:
- TPA Analysis: Diagnostic tests show that TPA reduces the student-to-prior KL divergence and lowers external perplexity, directly correlating with improved syntactic structure and reduced grammatical errors (e.g., fewer function-marker omissions).
- OTA Analysis: OTA significantly improves IDF-weighted recall for content words (actions, objects, attributes) and reduces lexical error rates (omissions, substitutions). Visual occlusion experiments confirm that OTA successfully identifies generation-sensitive visual evidence.
- Robustness: The method remains effective across different LLM backbones (NLLB, mBART, mT0).

Significance
The paper claims that DualAnchor demonstrates that preserving linguistic form and enforcing fine-grained visual grounding are distinct but necessary components for high-quality sign language translation. By decoupling these objectives into specific training constraints, the method allows LLM-based SLT to leverage the fluency of pretrained language models while correcting the lexical inaccuracies inherent in standard sentence-level alignment. The results suggest that targeted regularization (TPA) and granular alignment (OTA) are more effective than simply increasing model capacity or relying on generic contrastive learning.

The Two Big Problems: Losing the Plot and Mixing Up the Details

The Solution: DualAnchor

What They Found

Technical Summary: DualAnchor for Gloss-Free Sign Language Translation

More like this