💻 computer science

Next-Scale Autoregressive Models for Text-to-Motion Generation

The paper introduces MoScale, a next-scale autoregressive framework that generates text-to-motion hierarchically from coarse to fine resolutions with cross-scale and in-scale refinement mechanisms, achieving state-of-the-art performance, high training efficiency, and strong zero-shot generalization.

Original authors: Zhiwei Zheng, Shibo Jin, Lingjie Liu, Mingmin Zhao

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhiwei Zheng, Shibo Jin, Lingjie Liu, Mingmin Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to dance based on a written description. If you tell the robot, "Do two jumping jacks, then turn around, pick something up, and turn back," a standard AI might get the individual steps right but mess up the order or the count. It might do three jumping jacks, or forget to turn around at the end.

This paper introduces a new AI model called MoScale that solves this problem. Here is how it works, explained through simple analogies:

The Problem: The "One-Step-at-a-Time" Trap

Most current AI models for motion generation work like a person reading a book one word at a time. They predict the next movement based only on the one that came right before it.

The Analogy: Imagine trying to paint a massive landscape by only looking at the tiny brushstroke you just made. You might get the colors of that one spot right, but you lose the big picture. You might forget that the mountain needs to be on the left or that the river needs to flow in a specific direction.
The Result: The robot's movements look smooth locally (the knee bends correctly), but the whole story is wrong (it doesn't do the two jumping jacks you asked for).

The Solution: The "Architect to Interior Designer" Approach

MoScale changes the strategy. Instead of predicting one tiny movement at a time, it builds the motion in layers, from coarse (big picture) to fine (small details).

The Coarse Scale (The Architect): First, the model acts like an architect sketching the blueprint. It decides the entire structure of the dance at a low resolution. It says, "Okay, the whole sequence is: Jump, Jump, Turn, Pick, Turn." It locks in the global story before worrying about how the fingers move.
The Fine Scales (The Interior Designers): Once the blueprint is set, the model zooms in. It takes that rough sketch and adds the details, like exactly how high the jump is or how fast the turn happens. It refines the motion step-by-step, but it never changes the main story the architect already decided.

The Secret Sauce: Two "Refinement" Tricks

The authors added two special features to make this process even better, especially since there isn't a huge amount of dance data available to train on.

1. The "Practice with Mistakes" Trick (Cross-Scale Hierarchical Refinement)

The Problem: If you only ever practice with perfect instructions, you panic when you make a mistake.
The Analogy: Imagine a student learning to drive. If the instructor only lets them drive on perfect, empty roads, they will crash the moment they hit a pothole.
MoScale's Fix: During training, the model is intentionally given "corrupted" or imperfect blueprints. It has to learn to fix the mistakes the "architect" made and still produce a good dance. This teaches the model to be robust and recover from errors, ensuring the final motion stays true to the text even if the early steps were slightly off.

2. The "Second Look" Trick (In-Scale Temporal Refinement)

The Problem: Even after deciding the big picture, the model might get a specific detail wrong, like a hand twitching the wrong way.
The Analogy: Imagine writing an essay. You write a draft, then you go back and read it. You spot a typo or a clunky sentence and fix it.
MoScale's Fix: After the model makes a prediction for a specific layer, it gets to "look back" at what it just wrote. It identifies the parts it is unsure about and re-predicts them. This is like a second draft that polishes the motion to make it smoother and more realistic, without messing up the overall story.

Why This Matters

The paper claims that MoScale is the first to successfully combine the speed of "next-token" models with the ability to understand long, complex stories.

Better Storytelling: It follows instructions like "two jumping jacks" or "turn, pick up, turn" much better than previous models.
Efficiency: It trains faster than the popular "Diffusion" models (which are like trying to turn a block of noise into a picture by slowly erasing the noise).
Versatility: Because it understands the structure so well, it can also edit existing dances (like changing a walk to a run) without breaking the parts of the dance you didn't want to change.

In short, MoScale stops the AI from getting lost in the details and forces it to plan the whole dance first, ensuring the robot actually does what you asked it to do.

Technical Summary: MoScale – Next-Scale Autoregressive Models for Text-to-Motion Generation

1. Problem Statement

Text-to-motion generation aims to synthesize human motion that faithfully reflects text descriptions. While recent models (Diffusion, Masked Transformers, and Next-Token Autoregressive models) produce realistic local dynamics, they struggle to preserve global semantic structure. Specifically, existing methods often fail to align with complex text instructions involving repetition counts (e.g., "two jumping jacks") or sequential action patterns (e.g., "turn around, pick things up, and turn around").

The authors identify two primary causes for this limitation:

Diffusion and Masked Models: These models generate full-resolution sequences without causal ordering, relying on bidirectional refinement. Inferring global semantics in a single, undirected prediction step is inherently difficult, and subsequent refinement primarily enforces local coherence rather than altering the global structure established in the initial draft.
Next-Token Autoregressive (AR) Models: While successful in language modeling, next-token AR underperforms in motion generation. Human motion exhibits highly predictable short-term dynamics, allowing AR models to minimize loss by exploiting short-horizon predictability rather than learning global temporal structures. Furthermore, temporal convolution encoders in motion tokenization amplify local correlations, making long-range reasoning less necessary for the model, resulting in locally coherent but globally misaligned motion.

2. Methodology: MoScale

The paper proposes MoScale, a Next-Scale Autoregressive framework that replaces standard next-token prediction with a coarse-to-fine causal hierarchy.

2.1 Hierarchical Motion Representation

MoScale utilizes a Residual VQ-VAE to encode motion sequences into $K$ scales of discrete tokens.

Coarse Scales: Encode global structure.
Fine Scales: Represent residual details not captured by previous levels.
Tokenization: Given an input motion $m$ , an encoder produces latent features. These are downsampled to different temporal resolutions ( $L_1, \dots, L_K$ ). At each scale $k$ , the target feature is the residual of the original features minus the accumulated reconstruction of previous scales. This residual is quantized using a shared codebook to produce discrete tokens $z_k$ .

2.2 Next-Scale Autoregressive Generation

Instead of predicting one token at a time, the model predicts an entire group of tokens $z_k$ for a specific scale, conditioned on:

The previously generated coarser scales ( $z_1, \dots, z_{k-1}$ ).
The text embedding (derived from a pretrained T5 encoder).

The architecture employs a causal Transformer with scale-wise causal masking in self-attention to prevent information leakage from finer scales to coarser ones, ensuring a strict hierarchical causality. Cross-attention attends to raw text embeddings to maintain consistent semantic guidance.

2.3 Refinement Strategies

To address data scarcity and error propagation, MoScale introduces two refinement mechanisms:

Cross-Scale Hierarchical Refinement:
- Problem: Standard next-scale AR uses teacher forcing with clean ground-truth inputs, failing to learn how to recover from imperfect coarse-scale predictions during inference.
- Solution: During training, the model introduces perturbations to the tokens of the coarser scale ( $k-1$ ) by randomly replacing a subset of tokens with random codebook entries. The model at scale $k$ is then trained to predict the residual based on this corrupted input. This forces the model to learn adaptive refinements and correct errors propagated from earlier stages, reducing exposure bias.
In-Scale Temporal Refinement:
- Problem: Text-motion datasets are small compared to language corpora.
- Solution: Within each scale, the model employs a selective mask-and-repredict strategy. Using bidirectional attention (available within a single scale), the model identifies low-confidence tokens, masks them, and iteratively re-predicts them. This mimics diffusion-style iterative refinement but operates within the causal AR framework, improving local temporal coherence and fidelity without disrupting the global coarse-to-fine structure.

3. Key Contributions

Next-Scale AR Framework: Introduces a hierarchical causal structure for motion generation that overcomes the limitations of next-token AR (short-horizon shortcuts) and the global alignment issues of diffusion/masked models.
Refinement Mechanisms: Proposes Cross-Scale Hierarchical Refinement to handle error propagation and limited data, and In-Scale Temporal Refinement to enhance local coherence using bidirectional context within scales.
State-of-the-Art Performance: Demonstrates that MoScale achieves SOTA performance on standard benchmarks while maintaining high training efficiency.
Zero-Shot Generalization: Shows the framework's ability to generalize to diverse tasks (inpainting, outpainting, editing, continuous generation) without task-specific fine-tuning.

4. Experimental Results

The model was evaluated on HumanML3D and KIT-ML datasets.

Quantitative Performance: MoScale outperforms existing Next-Token AR, Diffusion, and Masked Transformer baselines.
- On HumanML3D, it achieves the best R-Precision (Top-1: 0.540) and MM-Dist (2.830), with competitive FID (0.046).
- It scales effectively with model size (from 4 to 18 scales/steps).
Qualitative Performance: Visualizations show MoScale successfully captures fine-grained semantics (e.g., specific repetition counts and complex sequences) where baselines fail.
Complexity Analysis: MoScale shows the largest relative gains over baselines on high-complexity text descriptions, validating the effectiveness of the hierarchical design for complex semantic instructions.
Efficiency: MoScale trains faster than diffusion models and comparable masked transformers. Inference is efficient (0.28s for full model, 0.08s for minimal configuration), significantly faster than diffusion-based methods (e.g., MotionDiffuse at 7.12s).
User Studies: Users prefer MoScale over baselines for both text alignment (71.5% votes) and motion quality (73.3% votes). In zero-shot editing tasks, it receives the highest preference scores for motion quality, text alignment, and consistency.

5. Significance and Claims

The paper claims that hierarchical causality is the primary driver of improved text alignment. By establishing a global semantic scaffold at the coarsest scale and progressively refining it, MoScale avoids the "short-horizon shortcut" of next-token AR and the "global structure instability" of bidirectional refinement models.

The authors emphasize that:

The hierarchical prediction path is the dominant factor in capturing detailed textual cues, outperforming bidirectional iterative refinement alone.
Temporal refinement contributes mainly to local smoothness and consistency but plays a secondary role in global text alignment.
The framework is robust in limited-data regimes, leveraging cross-scale perturbation to simulate diverse intermediate states.
MoScale provides a unified approach for both conditional (text-to-motion) and unconditional (editing, inpainting) tasks via its mask-and-repredict strategy, preserving unedited regions effectively.

The work suggests that integrating diffusion-style benefits (iterative refinement, bidirectional context) within a causal, hierarchical autoregressive framework offers a superior balance of training efficiency, global semantic alignment, and local motion quality for text-to-motion generation.