Technical Summary: UMo – Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars

Problem Statement

The generation of expressive, real-time co-speech avatars—digital characters that synchronize facial expressions and body gestures with speech—is critical for interactive media, gaming, and virtual production. However, existing approaches face a fundamental trade-off between quality, efficiency, and data availability:

• Data Scarcity: Most high-fidelity methods rely heavily on expensive, paired audio-motion datasets, leading to poor generalization and limited motion diversity in varied real-world scenarios.
• Representation Limits: While Large Language Models (LLMs) offer powerful cross-modal capabilities, their dense architectures often struggle to represent the high-dimensional, fine-grained coordination required for simultaneous facial and body motion without incurring prohibitive computational costs.
• Latency Constraints: Interactive avatars require strict real-time performance (low latency and high throughput). Existing diffusion-based models are often too slow due to iterative sampling, while standard autoregressive models may lose historical context or fail to maintain high-fidelity alignment when forced into real-time streaming constraints.

Methodology: UMo Architecture

The authors propose UMo, a unified sparse motion modeling architecture designed to achieve real-time, high-fidelity co-speech generation. UMo integrates text, audio, and motion tokens into a single autoregressive formulation, employing three core mechanisms to balance quality and efficiency:

1. Unified Autoregressive Formulation

UMo treats motion generation as a token prediction task within a unified framework.

• Tokenization: Audio is encoded via a HuBERT feature extractor with k-means quantization (50 tokens/sec). Motion is decomposed into four region-specific streams (face, hands, upper body, lower body) and quantized using VQ-VAE tokenizers (30 tokens/sec per region).
• Chunk-wise Decoding: To manage latency and memory, the model operates on fixed-size sliding windows. It uses a prefix of historical motion tokens (P=10 frames) and predicts a target window (N=5 frames) autoregressively.
• Hybrid Attention: Leveraging a Flan-T5-Base backbone, UMo employs bidirectional attention for the historical prefix (to fully comprehend context) and causal attention for the future target window (to ensure streaming capability).

2. Spatially Sparse Mixture-of-Experts (MoE)

To address the representational limitations of shared parameters across different body parts, UMo introduces a spatially sparse MoE framework.

• Routing: The dense Feed-Forward Networks (FFNs) in the T5 backbone are replaced with MoE layers. A router dynamically assigns tokens from specific body regions (face, hands, upper, lower) to one of four dedicated expert networks.
• Benefits: This design increases the total model capacity and allows for specialized learning of distinct motion distributions (e.g., subtle facial expressions vs. gross body movements) without increasing the per-token computational overhead during inference, as only one expert is activated per token.

3. Temporally Sparse Keyframe Framework

To overcome the computational bottleneck of dense frame-by-frame generation, UMo adopts a two-stage temporal sparsity strategy.

• Keyframe Prediction: The main LLM backbone predicts only sparse, critical keyframes (e.g., every 6th frame) rather than every frame. This significantly reduces the autoregressive decoding length and latency.
• Interpolation Network: A lightweight, separate interpolation network reconstructs the dense motion sequence between the predicted keyframes. This network is trained to fill in non-keyframe intervals conditioned on fixed keyframe anchors, ensuring smooth transitions and preserving fine-grained details without burdening the main LLM.

4. Training Strategy and Audio Augmentation

UMo utilizes a three-stage training recipe to stabilize multimodal optimization:

1. Pre-training: Joint modeling of motion, audio, and text alignment using keyframe-based sequences.
2. Post-training Stage 1: Instruction tuning with dense temporal supervision to refine audio-motion alignment.
3. Post-training Stage 2: Adaptation to the chunk-wise autoregressive inference paradigm.

• Audio Augmentation: To mitigate data scarcity and improve robustness, the authors employ a targeted audio augmentation strategy. They use a Text-to-Speech model to generate diverse vocal timbres for existing motion sequences, creating a many-to-one mapping that encourages the model to learn universal motion patterns rather than overfitting to specific speaker characteristics.

Key Contributions

The paper outlines four primary contributions:

1. Unified Architecture: A novel autoregressive framework that jointly models text, audio, and compositional motion tokens, enabling high-fidelity, real-time generation.
2. Spatial Sparsity: A MoE backbone that scales representational power for different body parts without increasing inference latency.
3. Temporal Sparsity: A keyframe-centric generation paradigm combined with a learned interpolation network, achieving real-time performance while preserving motion details.
4. Practical Training Pipeline: A multi-stage training recipe and targeted audio augmentation strategy that enhances generation quality, diversity, and robustness in co-speech scenarios.

Experimental Results

The authors evaluated UMo on the BEATv2 dataset (Speaker 2), comparing it against state-of-the-art methods including LOM, MIBURI, MambaTalk, and GestureLSM.

• Quality & Synchronization: UMo achieved a Fréchet Gesture Distance (FGD) of 5.107 and Beat Constancy (BC) of 7.955, outperforming most LLM-based methods and demonstrating superior alignment and realism compared to non-LLM baselines like EMAGE and MambaTalk.
• Real-Time Performance: UMo achieved 44 FPS with a Time To First Frame (TTFF) of 826 ms on an NVIDIA A100 GPU. While MIBURI has a lower TTFF, UMo offers significantly higher motion expressiveness and fidelity.
• User Study: In a blind user study involving 13 participants, UMo (with audio augmentation) received the highest preference scores (average rank 1.78) for smoothness, diversity, and overall preference, significantly outperforming baselines like LOM and GestureLSM.
• Ablation Studies: Experiments confirmed that the combination of AR, MoE, and keyframe strategies yields the best results. Specifically, the MoE framework improved diversity without computational cost, and the keyframe strategy reduced cumulative decoding errors.

Significance and Claims

The paper claims that UMo provides a practical solution for high-fidelity real-time co-speech avatars. By unifying multimodal data within a sparse, autoregressive framework, UMo successfully navigates the trade-off between the high computational cost of large models and the strict latency requirements of interactive applications.

The authors emphasize that their approach:

• Preserves fine-grained speech-motion alignment even under strict latency constraints.
• Offers a scalable path for utilizing massive, diverse human motion data without requiring expensive, perfectly paired audio-motion datasets for every scenario.
• Demonstrates that sparse modeling (both spatially via MoE and temporally via keyframes) is a viable and effective strategy for real-time generative avatar systems.

The work concludes that while limitations remain regarding structural biases in temporal decoding and the need for external audio inputs, UMo lays a solid foundation for future unified generation frameworks in digital avatars.