Think in Latent Thoughts: A New Paradigm for Gloss-Free Sign Language Translation

1. Problem Statement

Sign Language Translation (SLT) aims to convert sign language videos into spoken/written text. While recent advancements have moved toward gloss-free SLT (direct video-to-text without intermediate gloss annotations), existing methods suffer from a fundamental misconception: they treat sign video segments as directly mappable to static spoken-language words.

The authors argue that this assumption fails because sign language is a productive, context-dependent system. Meaning is often generated on-the-fly through spatial grammar, motion modulation, and classifiers rather than fixed lexical entries. Consequently, SLT is not merely a sequence transduction task but a cross-modal reasoning problem. Current models struggle to:

• Disambiguate signs that rely on context and space.
• Maintain coherence over long video sequences.
• Faithfully ground generated text in specific visual evidence without explicit gloss supervision.

2. Methodology: SignThought Framework

The authors propose SignThought, a reasoning-driven framework that introduces an explicit intermediate layer of Latent Thoughts to bridge continuous visual evidence and discrete text generation. The architecture consists of three main components:

A. Sign Encoder

• Takes pre-extracted visual features (from a pretrained Inception network) as input.
• Uses a Conformer-style encoder (combining self-attention and convolution) to model long-range dependencies and local motion patterns.
• Outputs a sequence of frame-level evidence tokens ($E$).

B. Latent Chain-of-Thought (CoT) Module

This is the core innovation. Instead of feeding raw video features directly to the decoder, the model distills them into an ordered sequence of $K$ latent thought states ($C$).

• Learnable Thought Slots: The module initializes $K$ trainable vectors representing empty reasoning states.
• Thinking Layers: These layers iteratively refine the thought chain ($L$ layers deep) through two mechanisms:
  1. Causal Self-Attention: Enforces an ordered chain where the $k$-th thought only attends to previous thoughts ($1 \dots k$), creating a coarse-to-fine semantic progression.
  2. Evidence-Routed Cross-Attention: To avoid attention diffusion, the module performs structured routing:
     • It first generates soft segment tokens ($S$) from the video frames.
     • It computes a similarity matrix between thoughts and segments.
     • It applies Sinkhorn normalization to create a soft binding matrix ($A$), ensuring a balanced and contiguous assignment of video segments to specific thoughts.
     • Each thought retrieves fine-grained evidence based on this routing, creating a compact, high-level semantic plan.

C. Dual-Stream Decoder (Plan-Then-Ground)

The decoder follows a "Plan-then-Ground" generation strategy to separate semantic decision-making from evidence retrieval:

1. Think Stream: The decoder first attends to the Latent Thought Chain ($C$) to determine what semantic content to express next (the plan).
2. Ground Stream: Based on the attention weights from the "Think" step, the decoder derives a temporal prior. It then attends to the original encoder features ($E$) to retrieve the specific visual evidence required to realize that plan.

• This decoupling prevents the model from hallucinating text without visual support and improves faithfulness.

D. Training Objectives

• Translation Loss: Standard cross-entropy loss between the generated text and the reference sentence.
• Structural Regularization:
  • Monotonicity Loss ($L_{mono}$): Penalizes thoughts attending to earlier evidence than previous thoughts, enforcing temporal order.
  • Contiguity Loss ($L_{cont}$): Encourages each thought to focus on a contiguous region of the video, preventing fragmented attention.

3. Key Contributions

1. Latent Thought Abstraction: Introduces an explicit, ordered intermediate semantic interface (Latent Thoughts) that bridges the gap between continuous visual streams and discrete reasoning primitives, replacing the flawed "direct mapping" assumption.
2. Plan-Then-Ground Decoupling: Proposes a novel decoding mechanism where the model first plans the semantic content via latent thoughts and then grounds it in visual evidence, significantly improving coherence and faithfulness.
3. Structured Routing: Utilizes Sinkhorn-normalized routing to assign video segments to thoughts in a balanced and contiguous manner, solving the issue of attention diffusion in long videos.
4. LC-HKSLT Dataset: Released a new large-scale, gloss-free Sign Language Translation dataset (1,311 hours, 432K clips) derived from Hong Kong broadcast news. It features realistic, noisy, sentence-level supervision without gloss annotations, filling a gap in existing benchmarks.

4. Experimental Results

The authors evaluated SignThought on five benchmarks: PHOENIX-2014T, CSL-Daily, How2Sign, OpenASL, and the new LC-HKSLT.

• Performance: SignThought achieved State-of-the-Art (SOTA) results across all datasets in the gloss-free setting.
  • On PHOENIX-2014T, it achieved 27.22 BLEU-4 and 54.50 ROUGE, surpassing the previous best (C2RL) by significant margins.
  • On How2Sign and OpenASL, it showed substantial gains (e.g., BLEU-4 improved from 9.37 to 13.39 on How2Sign).
  • On LC-HKSLT, it established a new baseline (21.15 BLEU-4), with further gains (30.22 BLEU-4) when utilizing the full dataset scale.
• Ablation Studies:
  • Removing the latent thinking module caused the largest performance drop, confirming the necessity of intermediate reasoning.
  • Disabling causal ordering or structured routing degraded performance, proving the importance of the chain structure and evidence allocation.
  • The "Plan-then-Ground" mechanism outperformed standard single-stream decoders.
• Qualitative Analysis: Case studies showed that SignThought preserves core semantics and handles fine-grained details (e.g., specific entities like "KFC" or duration "all day") much better than baselines, which often hallucinate or generalize incorrectly.

5. Significance and Future Work

• Paradigm Shift: The paper challenges the prevailing view of SLT as simple video-to-text conversion, reframing it as a cross-modal reasoning task. This aligns SLT with recent advances in Chain-of-Thought reasoning in Large Language Models (LLMs).
• Scalability: The proposed framework is highly scalable and does not rely on expensive, manually annotated glosses, making it suitable for real-world deployment where such data is scarce.
• Interpretability: While the thoughts are "latent" (hidden states), the structured routing and regularization make the reasoning process more interpretable and traceable than black-box attention mechanisms.
• Future Directions: The authors suggest future work could focus on making these latent thoughts more explicit (e.g., verbalized rationales) and extending the framework to broader sign languages and open-world scenarios.

In conclusion, SignThought represents a significant leap forward in sign language translation by introducing a reasoning-driven architecture that effectively handles the complexity, context-dependency, and spatial nature of sign language without relying on intermediate gloss annotations.