💻 computer science

OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data

This paper introduces OmniDance, a framework that leverages the newly constructed large-scale CIPE-Dance dataset and a specialized architecture to achieve state-of-the-art music-driven, text-driven, and multimodal dance video generation while preserving high visual fidelity and controllability.

Original authors: Kaixing Yang, Jiashu Zhu, Xulong Tang, Ziqiao Peng, Xiangyue Zhang, Chubin Chen, Puwei Wang, Jiahong Wu, Xiangxiang Chu, Hongyan Liu, Jun He

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kaixing Yang, Jiashu Zhu, Xulong Tang, Ziqiao Peng, Xiangyue Zhang, Chubin Chen, Puwei Wang, Jiahong Wu, Xiangxiang Chu, Hongyan Liu, Jun He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a dance video where a person moves perfectly to a song, looking just like a real human. Until now, AI has struggled to do this well. It could either make the person move to the music but look like a glitchy cartoon, or make them look realistic but move stiffly, ignoring the beat.

The paper introduces OmniDance, a new system that solves this by combining a massive new library of dance data with a smarter way of teaching the AI how to dance.

Here is how they did it, broken down into simple parts:

1. The Problem: The "Empty Dance Floor"

Think of existing AI dance tools as dancers trying to learn on an empty stage. They don't have enough good examples to study.

The Data Gap: There weren't enough high-quality dance videos on the internet organized in a way that an AI could easily learn from.
The Method Gap: The AI models that make videos were trained to listen to text (like "a person dancing") but didn't know how to listen to music (the rhythm and beat) without getting confused.

2. The Solution Part 1: Building a Massive Library (CIPE-Dance)

To fix the data problem, the researchers built CIPE-Dance.

The "Expert Filter" Pipeline: Imagine you are hiring a dance instructor to find the best videos from the internet. Instead of one person checking everything, they used a team of AI "experts" with different jobs.
- First, a "lightweight" expert quickly checks if a video is clear and high quality.
- Then, "heavier" experts check for specific dance problems, like: Is it just one person dancing? (No group dances allowed). Is the camera shaking too much? Is the person facing away?
The Result: They ended up with 300,000 high-quality dance clips (over 400 hours of video).
The "Choreography Notes": They didn't just dump the videos in a folder. They used AI to write detailed "dance notes" for every video, describing not just what the dancer is wearing, but how they are moving (e.g., "hip isolations," "playful intent," "rhythmic arm swings"). This gives the AI a rich vocabulary to learn from.

3. The Solution Part 2: The Smart Teacher (OmniDance)

To fix the method problem, they built OmniDance, which is like a dance teacher that knows how to teach three different types of classes at once:

Text-to-Video: Dancing based on a written description.
Music-to-Video: Dancing based only on a song.
Music + Text: Dancing based on both.

They used a clever Three-Step Training Strategy (Curriculum Learning):

Step 1 (The Warm-up): The AI first learns to dance just using text descriptions. This stabilizes its ability to look like a real human.
Step 2 (The Guided Practice): They introduce the music, but the AI still has the text to help it. It learns how the music fits with the text, like a student learning to follow a beat while still reading the sheet music.
Step 3 (The Solo Performance): Finally, the AI learns to dance using only the music, without needing text help.

4. The Secret Sauce: "Depth-Aware" Specialization

The paper describes a smart architectural trick called Music-Text Progressive Specialization.

The Analogy: Imagine building a house.
- Shallow Layers (The Foundation): The AI uses Text first to build the "structure" of the dance (the general style, the pose, the story). This is like laying the bricks.
- Deep Layers (The Decoration): As the video generation gets closer to finishing, the AI switches focus to Music. It uses the rhythm to add the "finishing touches" (the specific timing, the bounce, the energy).
Why this works: If you try to listen to the beat and read the instructions at the exact same time with equal intensity, you get confused. By letting text set the stage and music refine the details, the dance looks natural and hits the beat perfectly.

5. The Result

When they tested OmniDance, it outperformed all other current methods.

Visuals: The dancers look real, with consistent faces and clothes (no glitching).
Movement: The movements are expressive and fluid.
Rhythm: The dancer hits the drum beats and musical changes accurately.
Flexibility: You can give it just a song, just a text prompt, or both, and it works great in all three scenarios.

In short: OmniDance is a new AI system that learned to dance by studying a massive, carefully filtered library of real dance videos and by being taught in a step-by-step way that lets it understand both the "story" of the dance (text) and the "rhythm" of the dance (music) without getting confused.

Technical Summary: OmniDance

Problem Statement

Music-driven dance video generation aims to synthesize expressive human motion temporally aligned with music while maintaining high visual fidelity. Despite recent advancements in AI-generated content (AIGC), existing methods struggle to simultaneously achieve expressive motion and high visual quality. The authors identify two primary limitations hindering progress:

Dataset Limitations: There is a lack of large-scale, high-quality datasets and effective data collection pipelines specifically tailored for dance video generation. Existing datasets are often small-scale (e.g., FineDance contains only ~8 hours of motion) or lack specific choreographic annotations.
Methodological Limitations: There is an absence of principled, framework-level solutions for effectively integrating music as a complementary conditioning signal into Video Generation Foundation Models (VGFMs), which are typically trained without audio supervision. Existing approaches often rely on intermediate representations (2D keypoints or 3D SMPL) or fail to leverage the full potential of end-to-end diffusion models, resulting in limited motion expressiveness or degraded visual appearance.

Methodology: OmniDance

To address these challenges, the authors propose OmniDance, a unified framework designed to integrate music conditioning into a Text/Image-to-Video (TI2V) foundation model without compromising its original controllability or visual fidelity. The framework operates on three co-designed levels:

1. Dataset: CIPE-Dance

The authors introduce CIPE-Dance, the largest open-source dataset for dance video generation to date, comprising approximately 300,000 high-quality clips (over 400 hours).

Construction Pipeline: A "Progressive Expert-Based Data Collection Pipeline" is employed to filter web-crawled data from popular creators. It utilizes a progressive easy-to-hard strategy, starting with lightweight verification experts (Qwen3-VL-2B) for simple tasks and employing heavier filtering experts (Qwen3-VL-8B) to identify and eliminate complex failure modes (e.g., group dances, mirror reflections, scene instability).
Annotations: The dataset features Choreograph Informed Text Annotations generated by Qwen3-VL-8B across five dimensions: Body Dynamics, Choreographic Content, Expressiveness, Camera Presentation, and Overall Look.
Diversity: The dataset covers over 30 dance genres, diverse environments (studios, streets, homes), and varied performer demographics.

2. Model Architecture: Music-Text Progressive Specialization

Built upon the WAN2.2-TI2V-5B Diffusion Transformer (DiT) backbone, OmniDance introduces a depth-aware specialization mechanism:

Architecture: An additional music cross-attention layer is added to each DiT block.
Progressive Specialization: Motivated by the observation that shallow layers capture low-frequency global structures while deeper layers refine high-frequency details, the model employs a depth-aware scaling strategy. Text conditioning dominates in shallow layers to establish semantic layout, while the influence of the music branch (via a residual update) progressively increases in deeper layers to refine rhythm-aware motion dynamics. The scaling factor is defined as $(\ell/L)^\gamma$ , where $\ell$ is the layer index and $L$ is the total depth.

3. Training Strategy: Easy-to-Hard Curriculum Learning

To extend the TI2V capability to unified TI2V, Music-to-Video (MI2V), and Music-Text-to-Video (MTI2V) generation, a three-stage curriculum is adopted:

Stage I (Text-Adaptation Warm-up): The model is trained exclusively on TI2V to adapt the pretrained backbone to dance-specific textual descriptions while keeping the music branch frozen.
Stage II (Anchored Music Integration): A music branch is introduced, and the model is trained on both TI2V and MTI2V. This allows music conditioning to be learned under text guidance, preventing abrupt degradation of the pretrained TI2V capabilities.
Stage III (Modality Decoupling): The model is trained on all three tasks (TI2V, MTI2V, MI2V), enabling the generation of dance videos solely from music when text is absent.

4. Inference Strategy: Modality-Specialized CFG

The authors propose a Modality-Specialized Classifier-Free Guidance (CFG) strategy. Recognizing that text conveys global semantics and music provides temporal dynamics, the guidance weights are time-dependent:

Early Timesteps: Text guidance dominates to establish global semantics.
Later Timesteps: Music guidance becomes increasingly influential to refine rhythm-aware motion dynamics.
The guidance scales decay over time (Text: $5 \to 1$ ; Music: $4 \to 2$ ), ensuring a smooth transition from semantic layout to rhythmic refinement.

Key Contributions

CIPE-Dance Dataset: The introduction of the largest open-source dance video generation dataset (300k clips) with choreography-informed annotations, constructed via a novel progressive expert pipeline.
OmniDance Framework: A unified framework that successfully injects music conditioning into a TI2V foundation model through a three-level design (architecture, curriculum learning, and inference), supporting seamless switching between TI2V, MI2V, and MTI2V tasks.
State-of-the-Art Performance: Extensive experiments demonstrate that OmniDance achieves SOTA performance across all three generation tasks while exhibiting robust multimodal integration capabilities.

Experimental Results

Experiments were conducted on the CIPE-Dance dataset, evaluating Video Quality, Motion Quality, and Alignment.

TI2V Task: OmniDance outperformed strong baselines (CogVideoX1.5-5B, HunyuanVideo, WAN2.2-TI2V-5B) in video quality metrics (IQ: 67.74, AQ: 55.17) and motion fidelity (FIDk: 13.55).
MI2V Task: OmniDance achieved SOTA results in music-driven generation, surpassing fine-tuned talking-head models (Hallo2) and inference-only baselines (WAN-S2V, Echomimic-V3) in visual quality (IQ: 65.61) and beat alignment (BAS: 0.293).
MTI2V Task: The model demonstrated effective multimodal synergy, achieving high consistency (OC: 13.08) and maintaining strong beat synchronization (BAS: 0.287) while preserving identity and scene stability.
Ablation Studies: Removing the progressive specialization (MTPS) or the curriculum stages significantly degraded temporal smoothness and beat alignment, validating the necessity of the proposed architectural and training designs.

Significance and Claims

The paper claims that OmniDance represents a significant step forward in multimodal-driven dance video generation by solving the dual challenges of data scarcity and effective music integration. By leveraging a large-scale, high-quality dataset and a principled framework that respects the complementary roles of text (semantics) and music (dynamics), the authors demonstrate that it is possible to generate dance videos with both high visual fidelity and expressive, rhythm-aligned motion. The work unifies three distinct generation tasks into a single model, eliminating the need for separate generators. While the authors acknowledge that real-time generation is not yet supported, they suggest that combining their approach with distillation techniques offers a promising pathway for future efficiency.