💻 computer science

DiMo: Discrete Diffusion Modeling for Motion Generation and Understanding

DiMo introduces a unified discrete diffusion framework that extends masked modeling to bidirectional text-motion understanding and generation, enabling flexible quality-latency trade-offs and diverse motion tasks through iterative token refinement, residual vector quantization, and Group Relative Policy Optimization.

Original authors: Ning Zhang, Zhengyu Li, Kwong Weng Loh, Mingxi Xu, Qi Wang, Zhengyu Wen, Xiaoyu He, Wei Zhao, Kehong Gong, Mingyuan Zhang

Published 2026-02-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ning Zhang, Zhengyu Li, Kwong Weng Loh, Mingxi Xu, Qi Wang, Zhengyu Wen, Xiaoyu He, Wei Zhao, Kehong Gong, Mingyuan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, magical sketchbook. In the past, if you wanted to draw a person dancing based on a description, or write a story based on a dance video, you had to use two different sketchbooks, or one very rigid one that could only draw one stroke at a time, from left to right. If you made a mistake in the first stroke, the whole picture was doomed, and you couldn't easily go back and fix it without starting over.

DiMo is a new kind of "smart sketchbook" that changes the rules. It's a single system that can do two things at once:

Read a description and draw the motion (Text-to-Motion).
Watch a motion and write a description (Motion-to-Text).

Here is how it works, using some everyday analogies:

1. The "Blurry Photo" Game (How it Generates)

Most old methods work like a person writing a sentence one word at a time. Once they write a word, they can't change it. If they write "The dog ran," they can't easily go back and change "ran" to "jumped" without rewriting the whole sentence.

DiMo works differently. Imagine you have a photo of a dancer, but it's completely covered in static noise (like a blurry TV screen).

The Process: DiMo doesn't draw the dancer stroke-by-stroke. Instead, it looks at the whole blurry picture at once. It guesses what the dancer looks like, then makes a guess at what the next version should look like, and keeps refining the image over and over again.
The Magic: It can fix mistakes anywhere in the picture instantly. If the left arm looks weird, it can fix just the left arm without messing up the right leg. It does this by "denoising" the image in parallel steps, like sharpening a blurry photo until it's crystal clear.

2. The "Speed vs. Quality" Dial

Because DiMo refines the image in steps, you get to choose how fast you want the result.

Need it fast? You can stop the process early (say, after 5 steps). The dancer will look a bit rough, but you get the result instantly.
Need it perfect? You let it run for more steps (say, 30). The dancer will look incredibly realistic and smooth.
It's like a camera with a "Speed vs. Quality" dial. You can slide it to get exactly what you need for the moment.

3. The "High-Definition" Translator (RVQ)

To make the dancer look real, DiMo uses a special tool called Residual Vector Quantization (RVQ).

The Analogy: Imagine trying to describe a complex painting using only 10 colors. It would look blocky and ugly. Now imagine you have a palette with 1,000 colors, but you use them in layers. First, you lay down the big shapes (the body), then you add the muscles, then the tiny details like hair strands.
The Result: DiMo breaks the motion down into these layers. This allows it to capture the "coarse" movements (like walking) and the "fine" details (like a finger twitch) separately, resulting in much smoother, more realistic animations.

4. The "Smart Coach" (GRPO)

Sometimes, even if the motion looks good, it might not match the story you told it (e.g., the text says "jump," but the character "walks").

The Analogy: DiMo has a built-in "coach" (called GRPO). After DiMo makes a move, the coach checks: "Does this match the text?" If not, the coach gives a gentle nudge to the system to try again. Over time, DiMo learns to listen to the coach better, ensuring the dance matches the story perfectly.

What Can It Actually Do?

According to the paper, this single system is a Swiss Army knife for motion and text:

Text to Motion: You type "A person is doing a backflip," and it generates the video.
Motion to Text: You upload a video of a dance, and it writes a caption like "The dancer spins and jumps."
Fixing Mistakes: If you have a video with a missing part (like a cut in the middle), DiMo can fill in the gap without needing new instructions.
Correcting Captions: If you have a video and a caption that doesn't quite match, DiMo can fix the caption to describe what's actually happening.

The Bottom Line

The paper claims that DiMo is better than previous methods because it doesn't get stuck making one mistake at the beginning. It can look at the whole picture, fix errors anywhere, and let you choose how fast or how high-quality you want the result. It has been tested on standard datasets (HumanML3D and KIT-ML) and shows it can create high-quality dances and write accurate descriptions, all within one unified framework.

Technical Summary: DiMo – Discrete Diffusion Modeling for Motion Generation and Understanding

1. Problem Statement

Current approaches to bidirectional text–motion understanding and generation predominantly rely on autoregressive (AR) paradigms, such as GPT-style models that tokenize motion and decode sequences token-by-token. While these methods have unified interfaces for Text-to-Motion (T2M) and Motion-to-Text (M2T), they suffer from inherent limitations:

Latency Scaling: End-to-end latency scales linearly with sequence length due to sequential decoding.
Limited Editability: The unidirectional nature of AR decoding hinders global, multi-step corrections, making editing and completion tasks difficult.
Architectural Fragmentation: Unifying text-conditioned and text-free variants (e.g., motion completion) often requires separate engineering branches and specialized training tricks.

The paper posits that a unified framework capable of high-quality generation, low-latency decoding, editability, and multi-task unification remains an open challenge.

2. Methodology: DiMo Framework

DiMo (Discrete Diffusion Modeling) proposes a discrete diffusion-style framework that treats both text and motion as noisy sequences. Instead of sequential autoregression, DiMo employs iterative masked token refinement under a global context, enabling parallel denoising.

Core Components

Motion Tokenization (RVQ):
To improve motion representation fidelity, DiMo utilizes Residual Vector Quantization (RVQ). Continuous 3D motion sequences are discretized into multi-level tokens. This approach achieves lower quantization error at comparable bitrates compared to single-stage quantization, preserving both coarse body structure and high-frequency details.
Unified Architecture:
The model is built upon a BERT-based masked language model backbone, which naturally aligns with bidirectional denoising objectives. It integrates:
- Motion Encoder: Processes RVQ token streams using lightweight Transformer encoders for each residual layer.
- Motion Decoder: Uses parallel prediction heads to refine backbone features and output logits over the motion vocabulary across all RVQ levels simultaneously.
- Shared Embedding Space: Text and motion features are fused within a shared denoising framework.
Multi-Task Scheduling:
DiMo unifies three tasks within a single training loop via random sampling:
- Text-to-Motion (T2M): Recovering corrupted motion tokens from unmasked text prompts.
- Motion-to-Text (M2T): Translating reserved motion tokens into natural language descriptions (captioning).
- Motion-to-Motion (M2M): Self-supervised recovery of masked motion tokens without text input, enhancing motion prediction and completion capabilities.
Training and Optimization:
- Loss Function: The model is trained with a masked cross-entropy loss over a linear masking schedule, exposing the model to varying degrees of corruption.
- Inference Strategy: A confidence-guided progressive inference strategy is used. The model iteratively commits high-confidence tokens first and refines uncertain regions over $K$ denoising steps.
- Reinforcement Learning (GRPO): To enhance alignment and controllability, the framework incorporates Group Relative Policy Optimization (GRPO). This critic-free variant fine-tunes the model using task-specific rewards:
  - T2M Reward: Measures semantic correctness by comparing generated motion captions (via a frozen M2T branch) against ground truth in CLIP embedding space.
  - M2T Reward: Combines verb consistency, semantic similarity (CLIP), and a length penalty to prevent degenerate outputs.

3. Key Contributions

The paper outlines three primary contributions:

Paradigm Shift & Tunable Decoding: DiMo introduces the diffusion-LLM training mechanism to bidirectional text–motion modeling. Unlike strictly left-to-right decoding, the iterative masked refinement enables step-by-step self-correction, reducing sensitivity to early token mistakes. Crucially, the number of refinement steps ( $K$ ) provides a tunable quality–latency trade-off, allowing practitioners to select Pareto-optimal operating points.
Unified Capability & Extensibility: The framework inherently supports standard T2M/M2T, as well as text-free and text-conditioned motion completion and prediction. It also demonstrates robustness in motion-guided text correction and placeholder disambiguation without architectural changes.
Practical Effectiveness: By employing RVQ for tokenization and GRPO for alignment, DiMo achieves competitive results on HumanML3D and KIT-ML datasets, demonstrating strong motion quality and bidirectional understanding within a single model.

4. Experimental Results

Experiments were conducted on HumanML3D and KIT-ML datasets.

Text-to-Motion (T2M): DiMo achieves competitive FID scores (0.047 with GRPO), indicating high geometric and perceptual fidelity. While retrieval-based metrics (R@1) are competitive, the authors note these are highly dependent on the evaluator's training distribution.
Motion-to-Text (M2T): DiMo shows strong performance across text metrics (BLEU, ROUGE-L, CIDEr, BERTScore), suggesting the learned motion representations preserve rich, structured semantic information.
Quality-Latency Trade-off: Ablation studies on denoising steps ( $K$ ) reveal that reducing steps from 30 to 5 significantly cuts inference time with only modest quality degradation. Notably, even with fewer steps, DiMo outperforms existing methods in T2M FID with minimal latency.
Tokenization Depth: A 6-stage RVQ was found to be optimal, balancing reconstruction fidelity (MSE) with downstream task performance.

5. Significance and Claims

The paper claims that DiMo addresses the fundamental bottlenecks of autoregressive motion generation by leveraging the efficiency of masked refinement and parallel denoising. Its significance lies in:

Unification: Successfully unifying T2M, M2T, and M2M tasks without requiring separate models or specialized branches.
Flexibility: Offering a natural mechanism to trade off inference speed against generation quality by adjusting the number of denoising steps.
Robustness: Demonstrating that a single model can handle diverse tasks, including text-free motion completion and motion-guided text correction, thereby validating the efficacy of a unified semantic space for cross-modal modeling.

The authors conclude that DiMo provides a strong balance between motion realism and semantic consistency, offering a scalable and extensible paradigm for future motion systems.