Technical Summary: Corpus Augmentation for Sign Language Translation via LLM-Guided Video Stitching

Problem Statement

Sign Language Translation (SLT) aims to convert sign language video into spoken-language text. While large weakly-aligned datasets have enabled scale and gloss-free methods have reduced reliance on expert annotation, high-quality parallel sign video–text pairs for fine-tuning remain scarce. This scarcity limits generalization on long-tail vocabulary and unseen constructions. Existing data augmentation approaches face significant hurdles: feature-level stitching is tied to specific encoders, skeleton-level methods require rendering steps to be usable by RGB models, and realistic video generation remains an open challenge. Furthermore, recent controlled evaluations (e.g., Sincan et al.) suggest that many reported improvements in gloss-free SLT vanish under strict conditions, with the largest verified gain over a baseline being only +0.98 BLEU-4.

Methodology

The authors propose a corpus augmentation pipeline that generates synthetic RGB video–text pairs without requiring external video corpora, generative video models, or additional human annotation. The pipeline relies solely on an existing gloss-annotated training corpus and a Large Language Model (LLM). The process consists of three stages:

1. Per-Gloss Clip Extraction: Using CTC forced-alignment segmentation (based on Zuo et al.), the system extracts frame-level clips for individual glosses from existing training videos. This creates a "gloss-indexed" video database without manual timing annotations.
2. LLM-Guided Gloss–Sentence Generation: A corpus-anchored LLM generates novel gloss–sentence pairs. To prevent vocabulary drift and ensure domain consistency (German weather broadcasts), the LLM is provided with:
   • The complete Phoenix gloss vocabulary.
   • 500 randomly sampled in-corpus examples as context.
   • "Anchor" pairs selected via a least-used strategy to ensure coverage of the training distribution.
     Generated sequences are filtered to remove exact matches, subsequences of existing sentences, and duplicates, ensuring genuinely new linguistic content.
3. Video Assembly: Synthetic sequences are assembled by randomly sampling sentences from the generated pool and assigning them to clips.
   • Signer Selection: For LLM-generated sentences, a signer is sampled uniformly from those who have recorded clips for every gloss in the sequence.
   • Clip Assignment: Clips are selected randomly from available occurrences for the assigned signer.
   • Transition Handling: Transition frames are sampled from the empirical distribution of inter-gloss gaps in the real dataset. Notably, the authors explicitly exclude L2-based transition optimization (smoothing boundaries), finding it counter-productive.

The resulting synthetic data consists of standard RGB video–text pairs, making it architecture-agnostic and directly consumable by RGB-based SLT models or convertible to pose/feature representations.

Key Contributions

1. Scalable, Architecture-Agnostic Pipeline: A method for expanding SLT corpora using only existing gloss annotations and an LLM, producing synthetic RGB video–text pairs compatible with any downstream RGB-based architecture.
2. Significant Performance Gain under Controlled Conditions: Within the rigorous evaluation framework established by Sincan et al., the proposed augmentation achieves a +2.92 BLEU-4 improvement over the reproduced GFSLT-VLP baseline (21.38 → 24.30). This gain is achieved without altering the model architecture, training objective, or hyperparameters.
3. Verification of Gain Source: The authors demonstrate that the improvement is not an artifact of increased gradient steps. A "double-sampled" real data baseline (matched for computation) scored 21.17 BLEU-4, statistically indistinguishable from the real-only baseline, confirming the synthetic content itself drives the performance.
4. Counter-Intuitive Empirical Findings:
   • VLP Harm: Introducing synthetic stitched data during Vision-Language Pretraining (VLP) causes downstream performance to collapse, despite improving pretraining objectives. The authors suggest the visual encoder learns representations that fit synthetic discontinuities but fail to generalize to real continuous signing.
   • Transition Smoothing: Optimizing clip transitions for visual smoothness (via L2 distance) harms fine-tuning performance. The authors propose that abrupt boundaries may act as a form of implicit regularization, forcing the model to rely on core visual content rather than temporal boundary cues.

Results

• Primary Results (200 Epochs): The augmented model achieved 24.30 BLEU-4 and 46.09 ROUGE-L, compared to the baseline's 21.38 BLEU-4 and 41.69 ROUGE-L.
• Ablation on LLM Content: Replacing LLM-generated sentences with original Phoenix training sentences (while keeping random clip assignment) caused performance to drop to 20.04 BLEU-4, falling below the no-augmentation baseline. This confirms that novel linguistic content, not just visual variation, is the primary driver of the gain.
• Contamination Analysis: The synthetic gloss sequences showed no closer proximity to the development or test sets than real training sentences (lower max-BLEU-4 scores), ruling out evaluation leakage.

Significance

The paper claims that data-centric approaches, specifically the injection of linguistically diverse synthetic pairs, are a more effective lever for improving SLT than architectural tweaks or pretraining strategies that introduce visual discontinuities. By demonstrating a gain of nearly +3 BLEU-4 in a controlled setting where other methods failed to show significant improvement, the work highlights the potential of LLM-guided data generation to overcome the scarcity of high-quality parallel sign-language data. The findings regarding the detrimental effects of visual smoothing and VLP pretraining with synthetic data offer critical guidance for future augmentation strategies, suggesting that preserving "natural" discontinuities may be beneficial for model robustness.