Technical Summary: GestureFM

Problem Statement

Interactive digital humans and embodied conversational agents require speech-driven body motion generation that operates with low latency. Current high-quality co-speech generation methods, often based on diffusion models or large Transformer architectures, are primarily designed for offline, full-sequence inference. These approaches typically assume access to the entire audio sequence or utilize bidirectional temporal context, making them unsuitable for real-time interactive applications where motion must be generated before a complete utterance is available. The core challenge is to synthesize high-quality, expressive body motion from short audio buffers with minimal first-chunk latency while maintaining temporal continuity in a streaming setting.

Methodology

The authors propose GestureFM, a compact latent Flow Matching (FM) framework designed for low-latency, streaming co-speech body-motion generation. The system consists of three primary components:

1. Gesture Variational AutoEncoder (Gesture VAE)

To reduce the computational burden on the generative model, the authors compress high-dimensional motion data into a compact latent space.

• Input: 168-dimensional SMPL-X body pose (165-D rotations) and global translation (3-D).
• Architecture: A bidirectional Transformer encoder (6 layers, $d_{model}=512$) with a strided Conv1d layer performs 4× temporal downsampling, mapping 176 input frames to 44 latent tokens. The latent space has a dimension of $D_z = 16$.
• Decoding: A causal Transformer decoder reconstructs the motion. While the encoder is bidirectional (for offline training), the decoder and downstream generative model utilize causal attention to support streaming inference.
• Training Objective: The VAE is trained with a composite loss including pose reconstruction, translation reconstruction, velocity terms, and a KL divergence term ($\beta = 10^{-5}$).

2. Audio-Conditioned Causal Flow Matching

The generative model operates in the normalized latent space of the VAE.

• Audio Conditioning: Audio is processed via a Log-Mel spectrogram extractor (80 bands) and a lightweight CNN encoder to produce 44 temporal audio tokens.
• Flow Matching Transformer: A 6-layer Transformer (5.9 M parameters) learns a velocity field $v_\theta$ that transports a noise distribution $p_0 = \mathcal{N}(0, I)$ to the target data distribution.
• Causal Mechanism: The model employs a causal self-attention mask, ensuring each latent token attends only to itself and previous tokens, enabling streaming generation.
• Latent Normalization: A critical step involves normalizing the target VAE latents using per-dimension mean and standard deviation computed over the training set. This aligns the empirical latent distribution with the Gaussian prior used by Flow Matching, preventing samples from falling off the learned motion manifold.

3. Streaming Inference Pipeline

• Sliding Window: The system generates 1-second motion chunks (30 frames) from a 2-second available audio context.
• Overlap Blending: Consecutive chunks overlap by 25% (0.25s), with frames blended using linear interpolation.
• ODE Integration: Motion is generated by integrating the learned velocity field using the midpoint ODE method. The number of integration steps ($K$) is a tunable parameter for latency.
• Boundary Smoothing: To address velocity discontinuities at chunk boundaries, a lightweight 7-frame local smoother is applied post-overlap blending. This blends the decoded motion with a local linear interpolation without altering the sequence length or audio timing.

Key Contributions

1. Compact Latent Representation: The proposal of a 16-dimensional latent space that compresses 168-D SMPL-X pose and translation, enabling efficient generation.
2. Latent Normalization Strategy: The introduction of latent normalization for Flow Matching, which significantly improves generation quality by aligning the target distribution with the Gaussian prior.
3. Causal Streaming Pipeline: A rolling-window streaming architecture that demonstrates the ODE step count has negligible impact on motion quality (Fréchet Gesture Distance), allowing for the selection of $K=2$ as the optimal configuration for lowest latency.
4. Boundary Continuity Analysis: The characterization of chunk-boundary velocity discontinuities in streaming deployment and the demonstration that a lightweight 7-frame local smoother reduces velocity jumps by 85% without degrading beat alignment.

Experimental Results

The model was evaluated on the BEAT2 dataset (60 hours of SMPL-X motion capture from 25 speakers).

• Motion Quality (FGD): The latent-normalized GestureFM achieved an official EMAGE Fréchet Gesture Distance (FGD) of 0.272 on the test set. This represents a 30% improvement over the unnormalized variant (0.390) and significantly outperforms the reproduced EMAGE body-motion checkpoint (2.373).
• Beat Alignment (BA): The model matched ground-truth beat alignment with a score of 0.758 (offline) and 0.762 (streaming with smoother).
• Latency: With $K=2$ ODE steps, the system achieves a 22 ms first-chunk latency at a real-time factor (RTF) of 0.022×.
• Ablation Studies:
  • Latent Dimension: A 16-D latent space yielded better downstream generation quality than a 32-D space, despite the latter having better VAE reconstruction, suggesting the 16-D space acts as an implicit regularizer for causal flow matching.
  • ODE Steps: FGD remained stable (0.293–0.305) across all tested $K \ge 2$, confirming that increasing steps does not improve quality but linearly increases latency.
  • Smoothing: The 7-frame smoother reduced boundary velocity jumps from 0.700 to 0.105 (85% reduction) while maintaining stable BA.
  • Cross-Chunk Memory: An extension attempting to pass latent tails from previous chunks failed due to exposure bias, causing a 6.9× degradation in FGD.

Significance and Claims

The paper claims that GestureFM establishes a strong yet efficient baseline for low-latency body-motion generation. Its primary significance lies in demonstrating that high-quality, streaming-compatible motion synthesis is achievable without the heavy computational cost of bidirectional attention or iterative diffusion sampling.

The authors emphasize that latent normalization is essential for aligning generated samples with the VAE manifold, and that minimal ODE steps ($K=2$) are sufficient for high-quality output, making the system viable for real-time interactive applications. Furthermore, the study highlights the practical necessity of post-processing (local smoothing) to mitigate the artifacts inherent in chunk-independent streaming, while noting that more complex stateful mechanisms (like latent-tail memory) require careful handling of exposure bias to be effective.

The work concludes that while the current model focuses on body pose and translation (excluding facial expressions and hands), it provides a foundational framework for future holistic, low-latency digital human systems.