LiveGesture Streamable Co-Speech Gesture Generation Model

1. Problem Statement

Current co-speech gesture generation systems face a critical limitation: latency and lack of streamability.

• Offline Dependency: Most state-of-the-art (SOTA) methods (e.g., DiffSHEG, EMAGE, GestureLSM) are "offline," requiring access to the entire speech utterance (future audio) before generating gestures. This introduces high latency, making them unsuitable for real-time interactive applications like virtual avatars, telepresence, or VR.
• Causality vs. Coordination: Existing streaming attempts often fail to maintain strict causality (zero look-ahead) while ensuring holistic full-body coordination. Many models either treat body regions independently (losing inter-joint dependencies) or entangle all joints in a single model that struggles with real-time decoding.
• The Goal: The authors aim to build the first framework capable of generating diverse, beat-synchronous, full-body gestures in real-time with zero look-ahead, supporting arbitrary sequence lengths without buffering future audio.

2. Methodology: LiveGesture Framework

LiveGesture is built on two core modules: a Streamable Vector-Quantized Motion Tokenizer (SVQ) and a Hierarchical Autoregressive Transformer (HAR).

A. Streamable Vector-Quantized Motion Tokenizer (SVQ)

The SVQ converts continuous SMPL-X motion sequences into discrete, causal tokens.

• Asymmetric Architecture: It uses a bidirectional encoder (to learn rich dependencies during training) and a strictly causal decoder (for inference). This ensures the latent space is globally coherent while the decoding process remains streamable.
• Two-Stage Training:
  1. Pretraining: An asymmetric autoencoder is trained to reconstruct motion from latent sequences.
  2. Quantization: The encoder and decoder are frozen. A region-specific codebook and a lightweight projection head (MLP) are trained to quantize the continuous latents into discrete tokens. This prevents "token collapse" and ensures the discrete tokens align with the causal decoder's expectations.
• Output: Discrete, time-synchronous motion tokens for four body regions: Upper Body, Lower Body, Hands, and Face.

B. Hierarchical Autoregressive Transformer (HAR)

The HAR generates the gesture tokens autoregressively based on live audio.

• Streamable Audio Encoder: A causal 1D convolutional network processes incoming audio (mel-spectrograms) into audio tokens, ensuring no future information leaks into the current prediction.
• Region-eXpert Transformers (xAR):
  • Four independent causal Transformers (one per body region) model the specific motion dynamics of each region (e.g., hands move faster than the torso).
  • They use causal audio-motion cross-attention to align gestures with the current and past audio, and causal temporal self-attention for internal motion consistency.
• Causal Spatio-Temporal Fusion (xAR-Fuse):
  • A global fusion module that takes the hidden states from the frozen xAR experts.
  • It employs Pre-Infusion Adapters (PILOR) to align the independent expert outputs into a shared space.
  • It uses a causal fusion transformer with three attention mechanisms:
    1. Audio-Motion Cross-Attention: Reinforces beat/semantic alignment.
    2. Inter-Region Spatial Attention: Captures coordination (e.g., arm-torso coupling).
    3. Global Temporal Attention: Maintains long-term coherence.
  • The output is fed into region-specific token classifiers to predict the next SVQ token.

C. Training Strategy: Autoregressive Masking

To handle the "exposure bias" (where training uses ground truth history but inference uses self-generated history) and streaming noise, the authors introduce Hybrid Masking:

• Uncertainty-Guided Token Masking (UGM): During training, tokens with low prediction confidence are masked out based on a cosine schedule. The model learns to recover from imperfect histories.
• Random Region Masking (RM): Entire body regions are randomly dropped, forcing the fusion module to reconstruct missing motion using only audio and the remaining regions.
• Classifier-Free Guidance: Audio or text inputs are randomly dropped during training to enable the model to learn an unconditional prior, which is used at inference to sharpen the alignment between speech and gesture.

3. Key Contributions

1. First Zero-Look-Ahead Framework: LiveGesture is the first system to achieve fully streamable, zero-latency full-body gesture generation supporting arbitrary sequence lengths.
2. Streamable Motion Tokenizer (SVQ): A novel asymmetric tokenizer that learns a globally coherent latent space while enforcing strict causality for real-time decoding.
3. Hierarchical Architecture (HAR): A design that separates local region dynamics (via xAR experts) from global coordination (via xAR-Fuse), enabling both fine-grained expressiveness and holistic body coherence.
4. Robust Training: The introduction of uncertainty-guided and region masking strategies significantly improves robustness against prediction drift and streaming noise.

4. Experimental Results

Evaluated on the BEAT2 dataset (60 hours of SMPL-X motion), LiveGesture was compared against SOTA offline methods.

• Performance Metrics:
  • Beat Constancy (BC): LiveGesture achieved 0.794, the highest among all methods (surpassing offline SOTA like EMAGE at 0.772), indicating superior rhythm alignment.
  • Diversity: Achieved 13.91, the highest score, showing rich motion variation.
  • Fréchet Gesture Distance (FGD): Scored 4.57, competitive with the best offline method (GestureLSM at 4.25) despite lacking future context.
  • Latency: Achieves <50 ms per 200 ms audio chunk, with a First-Token Latency of 250 ms.
• User Study: In a blind study, LiveGesture was rated highest for Speech-Gesture Synchrony (4.3/5) and competitive for Realism and Smoothness compared to offline models.
• Ablation Studies: Confirmed that the fusion module, causal cross-attention, and uncertainty-guided masking are critical for performance. Removing the fusion module caused a massive drop in FGD (4.57 → 15.52).

5. Significance

• Enabling Real-Time Interaction: LiveGesture bridges the gap between high-quality gesture generation and real-time interaction, making it viable for virtual assistants, VTubers, telepresence robots, and AR/VR avatars.
• Paradigm Shift: It demonstrates that strictly causal models can match or exceed the quality of offline models if the architecture is designed correctly (hierarchical experts + robust masking).
• Scalability: The framework supports arbitrary sequence lengths, allowing for continuous, uninterrupted conversation without the need for chunking or re-initialization.

In summary, LiveGesture represents a major step forward in embodied AI, providing a practical, low-latency solution for generating natural, synchronized, and expressive full-body gestures in real-time.