Technical Summary: MotionMERGE

Problem Statement

Current motion-language models, while successful in unifying tasks like comprehension and generation, operate primarily at a coarse granularity. They treat human motion as a monolithic sequence described by concise textual summaries, lacking the ability to understand, edit, or reason about specific body parts or precise temporal segments. This limitation stems from two fundamental bottlenecks:

1. Model Limitations: Existing unified frameworks lack mechanisms to focus on localized motion patterns and fail to align language with fine-grained spatio-temporal details.
2. Data Limitations: Training data lacks fine-grained supervision, specifically part-level annotations and temporally localized corrective instructions. Furthermore, existing datasets for motion editing are limited to whole-body modifications and lack the structured reasoning data (Chain-of-Thought) necessary for complex, multi-step editing.

This granularity gap prevents the deployment of motion models in high-precision scenarios such as animation, rehabilitation, and human-computer interaction, where control over specific limbs and time intervals is required.

Methodology: MotionMERGE

The authors propose MotionMERGE (Multi-granular motion Editing, Reasoning, Generation, and Explanation), a unified framework designed to bridge the granularity gap. The framework integrates three synergistic innovations:

1. Unified Multi-Granular Architecture

MotionMERGE reformulates diverse motion tasks as conditional text generation within a single Large Language Model (LLM).

• Motion VQ-VAE: Continuous motion data is discretized into movement primitive tokens using a Vector Quantized Variational Autoencoder (VQ-VAE). This reduces computational costs and error accumulation.
• Motion-Aware Language Model: Built upon a T5 backbone, the model processes interleaved sequences of text and motion tokens. It utilizes a unified vocabulary that merges text tokens, motion tokens (from the VQ-VAE codebook), and special control tokens (e.g., <Motionless>, <SEP>, <Motion Tokens>).
• Granularity Control: By using specific prompts and special tokens, the model can handle tasks ranging from coarse-grained captioning to fine-grained, part-level editing and temporal grounding within a single architecture.

2. Reasoning-Aware Granularity-Synergy (RAGS) Pre-training

To address the misalignment between discrete motion tokens and multi-level semantics, the authors introduce RAGS, a pre-training strategy that employs joint supervision across four targeted objectives:

• Temporal-Aware Supervision: Anchors motion tokens and descriptions to precise time intervals to overcome the LLM's lack of temporal priors.
• Localized Alignment Supervision: Decomposes the alignment problem by training the model to map specific parts of detailed descriptions to corresponding local motion segments, building a "dictionary" of local text-motion mappings.
• Motion Coherent Supervision: Reframes generation as motion infilling by conditioning on sparse motion tokens, enhancing dependencies across the sequence.
• Motion-Grounded Chain-of-Thought (CoT) Supervision: Decomposes complex motion transformations into sequences of intermediate states. Unlike traditional text-based CoT, each reasoning step is paired with an aligned intermediate motion state, enabling interpretable, stepwise reasoning.

3. MotionFineEdit Dataset

To support the training of fine-grained editing and reasoning, the authors curate MotionFineEdit, a large-scale dataset comprising:

• Scale: Approximately 837K atomic editing triplets and 144K complex triplets.
• Granularity: Provides fine-grained spatio-temporal corrective instructions, including body-part level editing, explicit temporal intervals, and duration modifications.
• Motion-Grounded CoT: Introduces the first annotations that decompose complex edits into sequences of intermediate motion states, facilitating stepwise reasoning.
• Construction: Built via a scalable pipeline involving a text-driven fine-grained motion generator, rigorous automatic filtering, and human verification to ensure physical plausibility and instruction fidelity.

Key Results

Extensive experiments demonstrate that MotionMERGE outperforms state-of-the-art methods across both coarse and fine-grained benchmarks:

• Text-Driven Motion Editing: On the MotionFineEdit benchmark, MotionMERGE achieves a 79.84% R@1 for sequence-level retrieval and 41.81% for snippet-level (fine-grained) retrieval on atomic edits. In contrast, prior methods (e.g., MotionLab, TMED) perform near random chance (e.g., ~7% R@1) on snippet-level tasks, highlighting the granularity gap.
• Motion Captioning: On HumanML3D, MotionMERGE achieves state-of-the-art performance in R-Precision (Top-1: 0.602) and BERTScore (38.8). On the FineMotion dataset for detailed captioning, it consistently outperforms its predecessor (MG-MotionLLM) across all model sizes.
• Text-to-Motion Generation: MotionMERGE achieves the best performance among unified frameworks on HumanML3D, with an R-Top3 of 0.826 and FID of 0.233, narrowing the gap with specialized diffusion-based generators.
• Zero-Shot Reasoning: The model demonstrates emergent zero-shot capabilities in complex tasks, such as decomposing multi-step instructions (e.g., "step forward, bow, step back") into intermediate actions and generating corresponding motion scripts, validating the effectiveness of the CoT supervision.

Significance and Claims

The paper claims that MotionMERGE represents a significant step toward models that interact with motion in finer granularity and with human-like reasoning. Its significance lies in:

1. Unification: It is the first framework to unify diverse motion-language tasks (generation, understanding, editing, reasoning) across multiple granularities within a single model.
2. Bridging the Gap: By introducing RAGS pre-training and the MotionFineEdit dataset, it effectively addresses the dual bottleneck of insufficient localized modeling and lack of fine-grained supervision.
3. Paradigm Shift: The work challenges the specialization-generalization trade-off, demonstrating that investing in fine-grained understanding and data quality allows unified models to achieve performance comparable to specialized models while offering broader capabilities.
4. Benchmarking: MotionFineEdit establishes a new, challenging benchmark that reveals the limitations of existing methods, particularly in localized spatio-temporal control and complex reasoning.

The authors conclude that their approach enables a leap in language-guided motion control, interpretability, and reasoning, paving the way for more precise applications in animation and interaction.