Technical Summary: Coordinating Multiple Conditions for Trajectory-Controlled Human Motion Generation (CMC)

1. Problem Statement

Trajectory-controlled human motion generation aims to synthesize realistic human movements conditioned on both textual descriptions and spatial trajectories. While recent diffusion-based methods have attempted to integrate these conditions, the paper identifies two critical limitations in existing approaches (e.g., GMD, Omnicontrol):

1. Conflict Between Conditions: Existing methods often apply text and trajectory conditions simultaneously during the denoising process. The paper argues that injecting trajectory guidance into the diffusion process disrupts the original text-to-motion mapping learned during training. This interference causes the network to struggle with semantic consistency, leading to motions that are either semantically inconsistent with the text or fail to follow the trajectory accurately.
2. Inconsistencies in Redundant Representations: Many methods utilize redundant motion representations (including local joint positions, rotations, and velocities). However, trajectory guidance is typically applied only to local positions. This creates an inconsistency where the guided positions change while rotations and velocities remain unchanged (or are implicitly reconciled by the network). This leads to instability during the denoising process, particularly as the process nears completion, degrading control accuracy.

2. Methodology: The CMC Framework

The authors propose CMC (Coordinate Multiple Conditions), a decoupled framework that employs a "divide-and-conquer" strategy to separate trajectory control from motion generation. The framework consists of two cascaded stages:

Stage 1: Trajectory Control

• Goal: Generate a simplified representation of the pelvis and controlled joints that strictly adheres to the input trajectory while considering the text prompt.
• Simplified Representation: Unlike prior works using redundant representations, this stage uses a simplified representation containing only root angular/linear velocities and local joint positions ($ra, rx, rz, ry, jp$). This eliminates the instability caused by conflicting rotation/velocity components during guidance.
• Process: A diffusion model generates the initial simplified representation. Subsequently, classifier-guidance (using an L-BFGS optimizer) is applied to refine this representation against the spatial trajectory.
• Outcome: The positions of the pelvis and controlled joints are fixed and passed to the next stage.

Stage 2: Motion Completion

• Goal: Synthesize the full-body motion based on the text prompt, using the Stage 1 outputs as fixed partial observations.
• Process: A diffusion inpainting model generates the full-body motion (using the standard redundant representation for the final output). The local positions of the pelvis and controlled joints from Stage 1 are substituted into the input noisy tensors via a binary spatial-temporal mask.
• Key Feature: This stage relies solely on the text condition for generation. The trajectory guidance is removed to prevent interference with semantic motion generation, ensuring the network focuses on high-fidelity motion completion.

Selective Inpainting Mechanism (SIM)

To address the issue of overfitting caused by limited motion inpainting training data, the authors introduce SIM.

• Mechanism: During training, the model alternates between two modes with a 50% probability:
  1. Motion Inpainting: Training on data where partial observations (controlled joints) are masked.
  2. Text-to-Motion Generation: Training on standard text-to-motion data (non-inpainting).
• Purpose: The non-inpainting task acts as a regularizer, preventing the model from overfitting to partial observations and maintaining its ability to generate diverse, natural motions for unseen inputs.

3. Key Contributions

1. Identification of Limitations: The paper explicitly identifies the conflict between text and trajectory conditions and the instability introduced by redundant motion representations as fundamental barriers in current trajectory-controlled generation.
2. Decoupled Framework (CMC): A novel two-stage framework that separates trajectory control (using simplified representations) from motion completion (using text-only guidance), effectively mitigating condition conflicts.
3. Selective Inpainting Mechanism (SIM): A training strategy that alternates between inpainting and generation tasks to enhance the generalization of diffusion inpainting models.
4. State-of-the-Art Performance: The framework achieves superior results in both control accuracy and motion quality on standard benchmarks.

4. Experimental Results

The authors evaluated CMC on the HumanML3D and KIT-ML datasets, comparing against state-of-the-art methods like Omnicontrol, TLControl, and GMD.

• Control Accuracy: CMC significantly outperforms existing methods in trajectory error metrics. For example, on HumanML3D with pelvis control, CMC achieved an average error of 0.51 cm compared to 3.38 cm for Omnicontrol and 1.08 cm for TLControl. Similar improvements were observed for multi-joint control (e.g., pelvis + 5 end-effectors).
• Motion Quality: CMC achieves lower Fréchet Inception Distance (FID) scores and higher R-precision, indicating better motion realism and text alignment. In multi-joint control scenarios, CMC reduced the FID to 0.075 (vs. 0.218 for Omnicontrol).
• Stability Analysis: Visualizations of control error across denoising steps show that the simplified representation yields stable, low-error trajectories, whereas redundant representations exhibit oscillating errors that increase toward the end of the process.
• Ablation Studies:
  • Removing the two-stage decoupling (using a single-stage approach) significantly degraded both FID and control accuracy.
  • Passing all joints from Stage 1 to Stage 2 (instead of just controlled joints) increased FID and decreased R-precision, confirming that passing too much trajectory information creates text-trajectory conflicts.
  • SIM was shown to improve generalization, reducing the performance gap between models trained on ground-truth partial observations versus synthesized ones.

5. Significance and Claims

The paper claims that CMC effectively resolves the inherent trade-off between motion realism and control fidelity that plagues previous methods.

• Decoupling as a Solution: By separating the tasks, CMC allows the network to fully learn the text-to-motion mapping in the second stage without the disruption of trajectory gradients, while ensuring precise trajectory adherence in the first stage through simplified representations.
• Generalization: The use of SIM allows the model to generalize well to out-of-distribution trajectories and text prompts, a common failure point for models trained exclusively on inpainting tasks.
• Practicality: The framework supports various control modes, including single-joint, multi-joint, and keyframe control, demonstrating versatility in applications such as animation and gaming.

The authors conclude that while CMC significantly mitigates the conflicts in multimodal motion generation, limitations remain regarding extreme out-of-distribution trajectories (e.g., a pelvis moving 3 meters above ground), which the model struggles to interpret as it falls outside the training distribution. They suggest future work could focus on representing joint heights relative to the ground rather than absolute coordinates.