Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades

This paper proposes a two-stage cascaded framework that generates controllable complex human motion videos by first using an autoregressive model to synthesize 2D skeleton sequences from text descriptions and then employing a pose-conditioned diffusion model with adaptive layer fusion to render high-fidelity videos, supported by a new synthetic dataset designed to overcome limitations in existing benchmarks.

The Gist

Imagine you want to create a movie where a character performs a backflip, a cartwheel, or a martial arts kick. You want to type a simple sentence like "a person does a backflip," and have the computer generate a perfect, realistic video of it.

Currently, AI is great at making videos of people walking or dancing, but when it comes to complex, acrobatic moves, it often fails. The limbs might twist into impossible shapes, the person's clothes might change color mid-air, or the character might look like a different person entirely.

This paper introduces a new "two-step recipe" to solve this problem, along with a special training dataset to teach the AI how to handle these tricky moves.

Here is the breakdown using simple analogies:

The Problem: The "Vague Director" vs. The "Rigid Actor"

• The Vague Director (Text): If you just tell an AI, "Do a backflip," it's like a director giving a vague instruction to an actor. The actor knows what a backflip is, but they don't know exactly how fast to spin, where their hands should be at frame 10, or how to land. The result is often a wobbly, unrealistic mess.
• The Rigid Actor (Pose Control): To fix this, previous methods asked users to draw the skeleton of the person for every single frame of the video. This is like asking a director to draw a storyboard for every single second of a movie. It's too much work for humans!

The Solution: A Two-Stage "Director and Choreographer" Team

The authors propose a system that splits the job into two specialized roles, working together like a Director and a Choreographer.

Stage 1: The Director (Text-to-Skeleton)

Goal: Turn your words into a precise dance plan.

• How it works: You type "a person does a backflip." The AI doesn't try to draw the video yet. Instead, it acts as a Choreographer. It translates your words into a sequence of 2D stick-figure drawings (a skeleton) showing exactly how the joints should move over time.
• The Magic: It uses a "chain of thought" approach. It predicts the next joint position based on where the previous joints were, ensuring the movement flows logically (like a real human) rather than jumping around randomly.
• The Result: You get a perfect, mathematically correct "stick-figure movie" of the action.

Stage 2: The Actor (Skeleton-to-Video)

Goal: Turn the stick-figure plan into a realistic movie with a specific person.

• How it works: You give the system the stick-figure plan from Stage 1 and a photo of the person you want to appear (e.g., "Make it look like this guy in a red tie").
• The Innovation (DINO-ALF): This is the paper's secret sauce.
  • Old Way: Previous AIs used a "global" memory (like CLIP) to remember what the person looked like. It's like remembering "He is a guy in a red shirt." But when he spins fast, the AI gets confused and forgets the details, turning the red shirt blue or losing the tie.
  • New Way (DINO-ALF): This new method uses a "magnifying glass" approach. It looks at the reference photo in tiny, detailed patches (texture, fabric, specific body parts) and stitches them together dynamically. Even when the person is upside down or spinning, the AI knows exactly where the red tie is and keeps it sharp. It's like having a super-precise costume designer who never loses a thread, no matter how fast the actor moves.

The Missing Ingredient: The "Gymnastics Gym" Dataset

To train this system, the researchers needed a lot of examples of people doing flips and stunts.

• The Problem: Real-world videos of complex stunts are rare, hard to find, and often have copyright or privacy issues.
• The Fix: They built a virtual gym using the game engine Blender. They created 2,000 synthetic videos of different characters doing acrobatics in different environments.
• Why it matters: It's like building a flight simulator for pilots. You can crash the plane a thousand times in the simulator to learn how to fly, without ever risking a real crash. This dataset taught the AI how to handle extreme movements without needing real-world footage.

The Results: Why This Matters

When they tested this system:

1. Better Planning: The "Choreographer" (Stage 1) created stick-figure plans that were much more physically realistic than previous AI methods.
2. Better Acting: The "Actor" (Stage 2) kept the character's face, clothes, and details consistent, even during fast spins and flips.
3. No More "Glitchy Limbs": The videos didn't have the weird, broken arms or legs that plague current AI video generators.

In a Nutshell

Think of this technology as a smart animation studio.

1. You give it a script (Text).
2. A Choreographer writes out the exact dance moves (Skeleton).
3. A Special Effects Artist puts a specific actor into those moves, ensuring their clothes and face stay perfect even when they are doing a backflip (Video).

This makes it possible to generate high-quality, complex action videos just by typing a sentence, opening the door for better sports training, virtual movie pre-visualization, and fun avatar animations.

Technical Summary

Here is a detailed technical summary of the paper "Controllable Complex Human Motion Video Generation via Text-to-Skeleton Cascades."

1. Problem Statement

Current video diffusion models struggle to generate complex human motions (e.g., flips, cartwheels, martial arts) that involve large pose changes, rapid transitions, and frequent self-occlusions. The paper identifies two primary bottlenecks:

• Temporal Ambiguity of Text: Text-only conditioning (e.g., "a person does a backflip") is semantically clear but temporally vague. It fails to specify frame-wise joint trajectories or sub-motion timing, leading to implausible limb trajectories and temporal inconsistency.
• Limitations of Explicit Pose Control: While pose-conditioned methods improve controllability, they require users to supply complete skeleton sequences. Generating high-quality 2D poses for long, dynamic actions is time-consuming and requires specialized tools, limiting scalability.
• Appearance Drift: Existing pose-conditioned models often rely on CLIP-based embeddings for reference appearance. CLIP provides global semantic representations that lack fine-grained spatial details, causing identity loss, texture blurring, and clothing inconsistencies under large deformations and self-occlusions.
• Data Scarcity: There is a lack of public datasets specifically for complex acrobatic motions; existing benchmarks focus on repetitive activities (like dancing) or suffer from copyright/privacy issues.

2. Methodology

The authors propose a two-stage cascaded framework that decouples motion planning from appearance synthesis.

Stage 1: Autoregressive Text-to-Skeleton Generation

This stage translates natural language descriptions into 2D pose sequences.

• Architecture: An autoregressive Transformer model.
• Tokenization: Continuous 2D joint coordinates are discretized into bins ($K=256$) to form a vocabulary. The sequence is serialized in a frame-major, joint-minor order (all joints for frame $t$, then frame $t+1$) to ensure spatial coherence.
• Conditioning: Text prompts are encoded using a frozen CLIP text encoder and prepended as a conditioning prefix to the pose token sequence.
• Mechanism: The model predicts joint tokens autoregressively, conditioned on previous tokens and the text. This explicitly models long-range temporal dependencies and inter-joint coordination required for complex, non-repetitive actions.

Stage 2: Pose-Conditioned Video Diffusion

This stage synthesizes the video from a reference image and the generated skeleton sequence.

• Backbone: Built upon the pretrained Wan2.1 (14B) text-to-video diffusion model, adapted for pose-driven generation.
• DINO-ALF (Adaptive Layer Fusion): To solve appearance drift, the authors replace standard CLIP-based reference encoding with a novel DINO-ALF module.
  • It extracts patch descriptors from multiple layers of a frozen DINOv3 encoder.
  • Adaptive Fusion: It uses a cross-attention mechanism to aggregate features from different DINO layers. Early layers capture texture-rich local details (crucial for clothing and hands), while later layers provide view-invariant semantics.
  • Result: This allows the model to preserve fine-grained appearance details (e.g., tie patterns, hand shapes) even during large deformations and self-occlusions.
• Motion Encoding: The generated skeleton sequence is rasterized into a 2D pose-control video and encoded via a 3D CNN into spatiotemporally aligned motion tokens.
• Robustness Training: To handle errors from the first stage, the video model is trained with stochastic augmentations on the skeleton inputs (joint jitter, dropout, and temporal shifts).

Dataset Contribution

The authors constructed a synthetic dataset of 2,000 videos using Blender.

• Content: Diverse characters performing acrobatic and stunt-like motions (backflips, cartwheels, kicks).
• Advantages: Provides full control over appearance, motion, and environment; avoids copyright and privacy concerns associated with web-scraped data; fills the gap of complex motion data in existing benchmarks.

3. Key Contributions

1. Autoregressive Text-to-Skeleton Model: A novel approach to generating structured, editable 2D pose sequences from text, eliminating the need for manual pose generation while capturing complex temporal dependencies.
2. DINO-ALF (Adaptive Layer Fusion): A deformation-aware appearance conditioning mechanism that adaptively fuses multi-layer DINO features. It significantly outperforms CLIP-based conditioning in preserving identity and texture under extreme motion.
3. Complex Motion Synthetic Dataset: A release of 2,000 high-quality synthetic videos focusing on under-represented acrobatic actions, providing a new benchmark for the community.

4. Experimental Results

The framework was evaluated on the proposed synthetic dataset and the Motion-X Fitness benchmark.

• Text-to-Skeleton Performance:

  • Outperformed state-of-the-art baselines (HumanDreamer, T2M-GPT, MLD) on FID (255.19 vs. 322.16), R-Precision, and Diversity.
  • Qualitative results showed more physically plausible trajectories, especially during inverted phases (e.g., mid-air flips) where other models produced distorted limbs.

• Pose-to-Video Performance:

  • Achieved the best results on VBench-I2V metrics among all compared methods (HumanVid, MimicMotion, VACE, etc.).
  • Subject Consistency: 91.31 (vs. 88.31 for VACE).
  • Motion Smoothness: 97.39 (vs. 94.81 for VACE).
  • FVD (Fréchet Video Distance): 471.3 (lowest/best), indicating superior distributional similarity to real videos.
  • Qualitative: The model successfully preserved fine details (e.g., a red tie, hand shapes) and avoided the "hand/leg artifacts" common in baselines during complex spins and kicks.

• Ablation Studies:

  • Confirmed that DINO-ALF is superior to single-layer DINO or CLIP embeddings for appearance preservation.
  • Showed that stochastic augmentation during training is critical for robustness when using predicted (noisy) skeletons.
  • Demonstrated that deeper decoders and optimal tokenization granularity ($K=256$) improve motion coherence.

5. Significance

This work addresses a critical gap in generative AI: the ability to generate controllable, complex, and dynamic human motion without manual intervention.

• Practical Impact: It enables applications in sports content creation, virtual coaching, stunt pre-visualization, and avatar animation where high-fidelity, non-repetitive motion is required.
• Technical Advancement: By decoupling motion planning (text-to-skeleton) from appearance synthesis (skeleton-to-video) and introducing DINO-ALF, the paper solves the dual challenges of temporal ambiguity and appearance drift that have plagued previous diffusion models.
• Community Resource: The release of a dedicated synthetic dataset for complex motions provides a necessary benchmark for future research in this domain.

Limitations: The current framework is limited to single-person generation and does not model multi-person interactions. Additionally, extremely fast rotations may still cause blurring of high-frequency details like fingers.