Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Production

Technical Summary: Context-Aware Autoregressive Diffusion for Gloss-Wise Sign Language Production

Problem Statement
Current Sign Language Production (SLP) methods face significant challenges in generating natural, sentence-level sign motions. End-to-end approaches that generate entire sequences at once often suffer from temporal drift and hand motion blur as sentence length increases, leading to accumulated prediction errors and a degradation of critical semantic details like finger articulation. Conversely, methods that stitch together pre-composed gloss segments often fail to capture the natural coarticulation inherent in sign language—the phenomenon where the execution of one gloss influences the next. Existing gloss-wise retrieval methods lack explicit modeling of the kinematic and semantic coupling between consecutive glosses, resulting in abrupt boundary transitions, mismatched rotations, and limited motion variation.

Methodology: GARD Framework
The authors propose the Context-Aware Gloss-wise AutoRegressive Diffusion (GARD) model, a framework designed to generate sign motions gloss-by-gloss while explicitly modeling coarticulation.

1. Autoregressive Diffusion Process:

   • GARD treats the gloss as the fundamental semantic unit. It generates motion for the $n$-th gloss ($m_n$) sequentially, conditioning on the current gloss word ($g_n$) and the previous gloss word ($g_{n-1}$).
   • The model utilizes a Context-Aware Denoiser based on a Transformer Encoder. It takes a noisy motion sample and predicts the injected noise to reconstruct the clean motion.
   • Multi-Conditioning: The denoiser is conditioned on three vectors:
     • Semantic context of the current gloss ($\tau_{gloss}(g_n)$).
     • Semantic context of the previous gloss ($\tau_{gloss}(g_{n-1})$).
     • Kinematic context of the previous motion ($\omega_{motion}(m_{n-1})$).
   • Semantic contexts are extracted via a fine-tuned mBART model, while kinematic contexts are encoded using a trainable Transformer encoder.

2. Inter-Gloss Transition Guidance (IG-Guidance):

   • To address pose gaps at the boundaries between consecutive glosses, IG-Guidance applies gradient-based control during the diffusion sampling process.
   • It minimizes the rotational discrepancy between the first frame of the current gloss and the final frame of the previous gloss (the "kinematic hint").
   • Crucially, it utilizes Geodesic distance on the rotation manifold $SO(3)$ rather than Euclidean distance to measure rotational error, ensuring geometrically accurate alignment of joint rotations.

3. Global Motion Harmonizer (GM-Harmonizer):

   • While IG-Guidance aligns the boundary, the GM-Harmonizer propagates this alignment influence throughout the entire gloss motion sequence.
   • Implemented as a trainable copy of the main denoiser with a residual learning strategy, it receives the kinematic hint and refines the subsequent frames.
   • It uses zero-initialized linear layers to inject corrections hierarchically, ensuring that the natural flow established at the boundary is maintained across the entire gloss duration without disrupting the initial generation.

Key Contributions

• GARD Architecture: A novel diffusion-based framework that performs gloss-wise autoregressive generation, leveraging both semantic (linguistic) and kinematic (motion) contexts to model coarticulation.
• Coarticulation Strategies: The introduction of two specialized modules:
  • IG-Guidance: Aligns boundary poses using gradient-based optimization on rotation manifolds to ensure seamless pose consistency.
  • GM-Harmonizer: Refines the global motion sequence to propagate boundary alignment, ensuring temporally smooth and coherent dynamics.
• Performance: The model achieves state-of-the-art results in both linguistic accuracy and motion similarity, outperforming existing SLP methods.

Experimental Results
The authors evaluated GARD on two large-scale benchmarks: PHOENIX-2014T (German Sign Language) and CSL-Daily (Chinese Sign Language).

• Quantitative Performance: GARD achieved superior scores compared to state-of-the-art keypoint-based and 3D mesh-based methods.
  • On Phoenix-T, it reached a BLEU-4 of 13.16 and ROUGE of 37.47, with a DTW-Hand score of 1.21 (lower is better).
  • On CSL-Daily, it achieved a BLEU-4 of 12.73 and ROUGE of 36.76, with a DTW-Hand score of 1.56.
  • The significant improvement in DTW-Hand (Dynamic Time Warping for hand joints) indicates superior capture of fine-grained finger articulation, which is critical for semantic meaning in sign language.
• Ablation Studies:
  • Context: Combining both semantic and kinematic contexts yielded the highest performance, with semantic context providing the most substantial reduction in Word Error Rate (WER).
  • Modules: Removing either IG-Guidance or GM-Harmonizer resulted in performance drops. IG-Guidance was shown to fix spatial misalignment at boundaries, while GM-Harmonizer was essential for maintaining temporal smoothness in subsequent frames.
  • Geometry: Using Geodesic distance for IG-Guidance outperformed standard L2 distance.
• Qualitative Analysis: Visual comparisons demonstrated that GARD produces more natural finger movements and body transitions compared to models like SOKE and S-MotionGPT, which often suffer from averaged poses or abrupt transitions.

Significance and Limitations
The paper claims that GARD addresses the fundamental limitation of current SLP methods by explicitly modeling the dependency between glosses. By treating glosses as the primary unit and enforcing kinematic continuity through diffusion-based guidance, the model generates sign sequences that are both linguistically accurate and visually natural.

The authors acknowledge a primary limitation: the reliance on an autoregressive, iterative diffusion sampling process results in inference speeds that are currently challenging for real-time interaction. However, the work establishes a new benchmark for high-fidelity, context-aware sign language synthesis.