Technical Summary: InteractGesture

Problem Statement

Co-speech gesture generation has advanced significantly in producing realistic full-body motion synchronized with speaker audio. However, existing models primarily focus on unconstrained generation, lacking fine-grained spatial controllability of individual body joints. Practical applications—such as digital humans, virtual agents, and embodied assistants—require precise spatial editing (e.g., directing a wrist to a specific object, framing an item with both hands, or pointing toward an interface) while maintaining natural speech-gesture synchrony.

A critical technical challenge arises in continuous streaming generation. Standard inference processes audio in finite, overlapping chunks. In a strictly sequential approach, once a chunk is generated, it is frozen. This creates a dependency bottleneck: if a spatial constraint in a future chunk ($k+1$) requires an adjustment to the trajectory in a preceding chunk ($k$), the frozen state of chunk $k$ prevents this correction, leading to motion discontinuities and inconsistent control across chunk boundaries.

Methodology

The authors propose InteractGesture, a model-agnostic, inference-time framework for spatially controllable gesture generation. The method does not retrain the underlying generator but instead guides the sampling process of a pre-trained diffusion model.

Core Mechanism: Latent Guidance

InteractGesture treats the pre-trained gesture generator as a denoising process. At selected sampling steps, the predicted clean target latent estimate ($\hat{x}_0$) is decoded through a differentiable RVQ-VAE decoder into body poses (SMPL-X joints).

1. Loss Computation: A masked spatial loss is computed between the decoded joint positions and user-specified control targets (positions, trajectories, or directions).
2. Gradient Backpropagation: Gradients are backpropagated from the spatial loss through the differentiable decoder to update the target latent estimate ($\hat{x}_0$), rather than the model parameters.
3. Denoising Update: The adjusted latent estimate is used to update the current motion latent ($x_{i-1}$). Subsequent sampling steps allow the learned score network to regularize the motion back toward the speech-conditioned gesture distribution, balancing spatial constraints with naturalness.

Progressive Chunk Guidance

To address the streaming dependency problem, the authors introduce Progressive Chunk Guidance. Unlike standard sequential inference (which freezes past chunks) or fully synchronous inference (which requires offline access to the entire sequence), this strategy maintains an active window of editable chunk latents.

• Staggered Delays: New chunks are introduced with a staggered delay relative to the current sampling step.
• Bidirectional Propagation: As new chunks arrive, preceding chunks remain in an active, editable state. Spatial constraints from future chunks can propagate gradients backward across chunk boundaries to adjust preceding trajectories.
• Context Refresh: The context frames for a new chunk are continuously refreshed from the latest guided estimates of preceding chunks, ensuring smooth motion continuity while adhering to a fixed streaming latency budget.

Control Representation

The system supports diverse control modes via a target position tensor ($H$) and a binary mask ($M$):

• Sparse Joint Position Control: Directing specific joints to point at objects at designated frames.
• Dense Joint Trajectory Control: Guiding joints along continuous paths.
• Pointing Direction Control: Orienting limbs toward target directions.

Key Contributions

1. InteractGesture Framework: A model-agnostic inference-time method enabling fine-grained spatial control for pre-trained co-speech gesture generators without modifying the generator's architecture or training objective.
2. Progressive Chunk Guidance: A novel chunk-window sampling strategy that resolves the conflict between streaming generation and global spatial consistency, allowing future constraints to influence preceding motion without requiring full-sequence offline processing.
3. Comprehensive Evaluation: An extensive evaluation on the BEAT2 dataset demonstrating the method's ability to support sparse positioning, dense trajectory tracking, and directional pointing while preserving gesture quality.

Experimental Results

The method was evaluated on the BEAT2 dataset using GestureLSM as the base generator. Key findings include:

• Spatial Accuracy: InteractGesture with Progressive Chunk Guidance achieved an Average Control Error of 6.34 cm and a Location Error of 16.1% (error > 10cm). This significantly outperforms:
  • Inverse Kinematics (IK): High error (19.97 cm) and unnatural poses (FGD 1.043) due to lack of generative regularization.
  • Chunk-Wise ControlNet: High error (46.16 cm) due to limited access to neighboring context.
  • Sequential Chunk Guidance: Moderate error (11.70 cm) due to frozen past chunks preventing backward adjustment.
• Gesture Quality: The method preserves speech-conditioned naturalness, achieving a Fréchet Gesture Distance (FGD) of 0.431 and Beat Consistency (BC) of 0.762, comparable to the Ground Truth and superior to post-hoc geometric fitting methods.
• Ablation Studies:
  • Full Guidance: Combining sampling guidance (during denoising) and post-refinement yielded the best results. Sampling guidance provided coarse alignment, while post-refinement fine-tuned the final latent estimates.
  • Scale Sensitivity: A guidance scale of 0.5 offered the optimal trade-off between spatial precision and gesture naturalness in the streaming setting.
  • Streaming vs. Offline: While the offline Synchronous Chunk Guidance achieved the highest precision (4.67 cm error), Progressive Chunk Guidance maintained comparable accuracy (6.34 cm error) while enabling continuous streaming execution.

Significance

The paper claims that InteractGesture bridges the gap between high-quality, speech-synchronized gesture generation and the practical need for precise spatial interaction. By enabling fine-grained multi-joint 3D spatial control during continuous streaming, the method supports complex embodied interactions (e.g., pointing, object manipulation) that were previously unattainable with unconstrained generative models. The Progressive Chunk Guidance mechanism is highlighted as a critical innovation, reconciling long-horizon controllability with the constraints of online, streaming inference.