Not All Frames Are Equal: Complexity-Aware Masked Motion Generation via Motion Spectral Descriptors

1. Problem Statement

Current masked generative models for text-to-motion synthesis (e.g., MoMask, MMM) suffer from a fundamental limitation known as the "Dynamic Uniformity Assumption."

• The Issue: These models treat all motion frames uniformly during masking allocation, attention mechanisms, and iterative decoding. They apply the same modeling budget regardless of the local difficulty of the motion.
• The Reality: Motion dynamics are highly non-uniform. Static poses or smooth transitions are easy to infer from context, whereas frames involving kicks, jumps, spins, or rapid phase transitions are dynamically complex and harder to reconstruct.
• The Consequence: Because standard models do not distinguish between easy and hard frames, reconstruction errors concentrate disproportionately on dynamically complex regions, leading to degraded motion quality in high-dynamic sequences.

2. Methodology: DynMask

The authors propose DynMask, a framework that relaxes the dynamic uniformity assumption by introducing a motion-grounded, parameter-free signal called the Motion Spectral Descriptor (MSD).

A. Motion Spectral Descriptor (MSD)

MSD is a deterministic, interpretable measure of local dynamic complexity computed directly from the motion signal, requiring no learned predictors.

• Computation:
  1. Velocity Extraction: Compute the temporal first difference of token embeddings ($v_t = x_t - x_{t-1}$).
  2. Sliding-Window DCT: Apply a Type-II Discrete Cosine Transform (DCT) to a local window of velocity vectors.
  3. Descriptor Generation: Aggregate the spectrum across feature dimensions to form a vector $\phi_t$ (spectral profile) and a scalar $\Omega(t)$ (complexity summary).
• Utility:
  • Scalar ($\Omega$): Ranks frames by dynamic difficulty.
  • Vector ($\phi$): Enables pairwise comparison of local motion dynamics (spectral similarity) for attention mechanisms.

B. Core Model Components

DynMask integrates MSD into the training and architecture of the masked transformer:

1. Complexity-Aware Mask Selection (Content-Focused Selection):
   • Instead of random masking, the model allocates the masking budget based on a dual-quota strategy.
   • A portion of masks is assigned to frames with high dynamic complexity (high $\Omega$) to force the model to learn difficult reconstructions.
   • Another portion is assigned to frames with high semantic salience (alignment with text) to preserve semantic fidelity.
2. Motion-Aware Attention:
   • Standard self-attention is augmented with a spectral similarity prior.
   • The attention logits are blended with a similarity score derived from the MSD vectors ($\phi_i$ and $\phi_j$).
   • This biases information flow toward frames with compatible dynamic structures (e.g., similar gait phases), even if they are temporally distant, improving the modeling of rhythmic and repetitive motions.

C. Full Model Extension: Complexity-Aware Decoding

During inference (iterative decoding), DynMask uses MSD to modulate the sampling process:

• Exploration Score: Combines dynamic complexity ($\Omega$) and model uncertainty.
• Adaptive Sampling: Frames that are dynamically complex or uncertain are assigned higher sampling temperature and noise, encouraging broader exploration. Simple, confident frames are decoded more conservatively.

3. Key Contributions

1. Identification of Limitation: The paper formally identifies the "dynamic uniformity assumption" as a systematic bottleneck in current masked motion generation, showing that uniform treatment fails to capture sharp variations in local motion difficulty.
2. Motion Spectral Descriptor (MSD): Introduction of a novel, parameter-free, and deterministic signal derived from the short-time spectrum of motion velocity. It provides both a scalar complexity ranking and a structured spectral profile for frame comparison.
3. DynMask Framework: A unified approach that reuses the MSD signal across three stages:
   • Training: Guiding content-focused mask selection.
   • Architecture: Enabling motion-aware attention.
   • Inference: Modulating complexity-aware decoding.
4. State-of-the-Art Performance: Achieving superior results on standard benchmarks, particularly for dynamically complex motions.

4. Experimental Results

The method was evaluated on HumanML3D and KIT-ML datasets.

• Quantitative Performance:
  • FID (Fréchet Inception Distance): DynMask achieved the best FID among masked methods, reducing HumanML3D FID from 0.045 (MoMask) to 0.028 (a 38% relative improvement).
  • Alignment & Diversity: It also achieved the best Multimodal Distance (MM-Dist) and Diversity scores.
  • Complexity Stratification: The improvements were most significant for dynamically complex motions (e.g., kicks, jumps), where the baseline FID was 0.227 and DynMask reduced it to 0.158.
• Ablation Studies:
  • Removing MSD (uniform masking) resulted in significantly worse performance.
  • MSD outperformed other complexity signals like joint velocity magnitude or loss-based hardness proxies.
  • Both the mask selection and attention components provided non-redundant gains.
• Cross-Dataset Transfer: When applied zero-shot to the BABEL dataset, DynMask improved FID by 22.5%, demonstrating that the complexity prior generalizes beyond the training distribution.
• Qualitative & User Study:
  • Visualizations showed DynMask better preserved dynamic timing, limb extension, and phase continuity.
  • A user study with 37 participants showed DynMask was preferred over MoMask and BAMM in Motion Quality (55.4%), Text Faithfulness (67.9%), and Temporal Naturalness.

5. Significance

This work establishes that respecting local motion complexity is a critical design principle for masked generative models. By moving away from uniform processing and utilizing a simple, physics-grounded spectral descriptor, DynMask demonstrates that:

• Masked models can be made "complexity-aware" without introducing heavy auxiliary models or learnable difficulty predictors.
• The mismatch between uniform masking strategies and non-uniform motion dynamics is a primary source of error in current state-of-the-art systems.
• The proposed approach offers a scalable, interpretable, and effective path forward for high-fidelity 3D human motion synthesis.