🤖 AI

MusicInfuser: Making Video Diffusion Listen and Dance

MusicInfuser is an efficient method that adapts pre-trained text-to-video diffusion models to generate high-quality, music-synchronized dance videos by selectively fine-tuning specific layers, achieving strong generalization and synchronization without requiring motion data or extensive computational resources.

Original authors: Susung Hong, Ira Kemelmacher-Shlizerman, Brian Curless, Steven M. Seitz

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Susung Hong, Ira Kemelmacher-Shlizerman, Brian Curless, Steven M. Seitz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a talented, world-class dance instructor who has spent years watching millions of videos. This instructor knows how to move, how to look good, and how to follow a text description like "a dancer in a chef's uniform." However, there's a catch: this instructor is completely deaf. If you play them music, they just keep dancing to their own internal rhythm, completely ignoring the beat.

MusicInfuser is the solution to this problem. It's a new computer program that teaches this "deaf" video-making AI to listen and dance to music, without having to hire a new teacher or start from scratch.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Deaf" Artist

Current video generators (like the one called Mochi) are amazing at creating visuals based on text. If you say, "A dog dancing on a beach," it can do it. But if you add music, the video doesn't sync up. The dog might be doing a slow waltz while the music is a fast-paced rock song.

Previous attempts to fix this tried to build a brand-new AI from the ground up that understands both sound and sight. But this is like trying to build a new orchestra from scratch when you already have a world-class one sitting in the room. It's expensive, slow, and often results in jerky, unnatural movements because there isn't enough data to teach it perfectly.

2. The Solution: The "Ear Implant"

Instead of building a new AI, MusicInfuser takes the existing, high-quality video AI and gives it "ears."

The Zero-Initialized Module: Imagine you are teaching a student to play the piano. If you give them a brand new, heavy set of keys, they might get overwhelmed and play the wrong notes immediately. Instead, MusicInfuser attaches a special "ear" to the AI that starts out completely silent (zero-initialized). At first, the AI ignores the music and just dances to its own rhythm. As it trains, this "ear" slowly wakes up and starts whispering instructions to the AI: "Hey, the beat just dropped, spin faster!" or "The music is soft, move gently."
The "Smart" Placement: The researchers didn't just attach these ears everywhere. They figured out exactly where in the AI's brain the music should connect. They used a special test to find the layers of the AI that are most sensitive to movement and structure, and they plugged the music in there. This is like tuning a radio to the exact frequency where the signal is clearest, rather than blasting noise through every speaker.

3. The Training: Learning from the Wild

Usually, dance videos used for training are very stiff and filmed in a studio with a white background. It's like learning to dance only in a gym. MusicInfuser also learned from "in-the-wild" videos—real dance clips from YouTube with different lighting, cameras, and messy backgrounds. This helped the AI learn that a dancer can look good even if the camera is shaky or the lighting is weird.

4. The Results: Listening and Dancing

The result is a system that can take a text prompt (e.g., "A female dancer in a Hawaiian dress on a beach") and a music track, and generate a video where:

The Moves Match the Beat: The dancer kicks, spins, and jumps exactly when the music hits a drum or a melody.
The Style is Flexible: You can change the text to make the dancer wear a tuxedo or dance in a kitchen, and the music will still sync perfectly.
It's Fast and Cheap: Because they didn't have to rebuild the whole AI, they could train this "listening" ability on a single computer card in less than a day.

What It Can and Cannot Do

It Can: Generate diverse dance moves for unseen music (even new genres like K-pop), create videos of animals dancing, and handle long videos. It creates realistic details like hair moving and clothes flowing, which older "skeleton" methods (which just draw stick figures) miss.
It Cannot: It still inherits some quirks from the original video AI. Sometimes, if the dancer moves very fast, the AI might get confused about fingers or faces, or it might swap the position of body parts. It is also limited by how good the original video AI was at looking realistic.

In short: MusicInfuser is like taking a brilliant visual artist who can't hear, giving them a pair of high-tech headphones, and teaching them to dance to the rhythm of the music, all while keeping their original artistic style intact.

Technical Summary: MusicInfuser

Problem Statement

Current open-source video diffusion models predominantly generate silent videos or those focused on speech, lacking the ability to synchronize motion with specific music tracks. While some research explores audio-video generation, training such models from scratch for the specific domain of dance is challenging due to the scarcity of high-quality, music-aligned dance datasets. Existing dance generation methods often rely on intermediate representations like skeletal motion or motion capture data. These approaches suffer from limitations: they are costly to acquire, prone to floating/jitter artifacts, and underparameterized for dance, failing to capture nuances such as backbone curvature, axial rotation, hand articulation, hair dynamics, and clothing motion. Furthermore, generating motion that is properly synchronized with a specified music track while maintaining control over style and appearance remains difficult.

Methodology

MusicInfuser proposes an approach to align pre-trained text-to-video diffusion models with musical inputs without requiring motion capture or reconstruction. The method adapts existing video diffusion models (specifically Mochi) to generate high-quality dance videos synchronized with specified music tracks, while preserving the model's ability to accept style and scene control via text prompts.

Core Architecture and Adaptation

The framework injects zero-initialized music-video modules into the Diffusion Transformer (DiT) blocks of the pre-trained model.

Zero-Initialized Cross-Attention (ZICA): To condition the model on audio, ZICA blocks are employed. The output projection matrix ( $W_O$ ) is initialized to zero, ensuring the cross-attention block acts as an identity function at the start of training. This prevents bias and destabilization, allowing the model to gradually incorporate audio features as $W_O$ learns during training.
Low-Rank Adaptors (LoRA): LoRA is used for domain and motion adaptation. Unlike standard image models, the authors employ higher-rank configurations (e.g., rank 64) to adequately capture the temporal dependencies required for complex human motion.

Layer Selection Strategy

Recognizing that fine-tuning all layers is computationally expensive and can degrade the pre-trained model's denoising capabilities (especially with limited specialized datasets), the authors propose a layer-wise adaptability criterion.

Instead of exhaustive search or intuitive selection (e.g., evenly distributed layers), they use a guidance-inspired constructive influence function.
This metric quantifies a layer's positive influence by performing guided sampling where the layer is skipped but used to provide guidance (similar to perturbed-attention guidance).
Layers that exhibit greater performance when excluded and used for guidance are identified as more intrinsically connected to structural and perceptual quality. These are the layers selected for adaptation, preserving the pre-trained knowledge while establishing correlations between music and movement.

Training Strategies

Beta-Uniform Scheduling: To preserve the pre-trained model's denoising capability, the training noise distribution evolves from a Beta distribution (concentrated on low noise levels) to a uniform distribution. This allows the model to first learn task-specific fine components (dance details) before adjusting the fundamental structure of movements, preventing the loss of general physics knowledge.
Data Utilization: The model is trained on a mixture of the constrained AIST dataset and "in-the-wild" data from YouTube. The in-the-wild data introduces diversity in camera trajectories, lighting, and environments, serving as regularization to prevent overfitting to specific studio settings.
Prompt Diversification: To reduce dependence on textual cues and encourage stronger music-motion associations, a portion of detailed captions is randomly replaced with basic prompts during training.

Key Contributions

Efficient Adaptation of Video Diffusion Models: Demonstrates how pre-trained text-to-video models can be efficiently adapted to musical inputs without training a multimodal model from scratch or using intermediate motion data.
Layer-Wise Adaptability Criterion: Introduces a novel metric based on a constructive influence function to select optimal layers for cross-attention adaptation, significantly reducing training costs while preserving rich prior knowledge.
Zero-Initialized Modules: Proposes ZICA and high-rank LoRA to stabilize training and allow gradual integration of audio features without destabilizing the pre-trained denoising manifold.
Training Efficiency: Achieves training on a single GPU within a day (approx. 20 hours for 4,000 steps) using a single NVIDIA A100.
Evaluation Framework: Introduces an automatic evaluation framework using Video-LLMs (VideoLLaMA 2 and Qwen3-Omni) to jointly assess video, audio, and language alignment, which correlates well with human judgment.

Experimental Results

Performance: MusicInfuser outperforms baseline models (including MM-Diffusion and Mochi) in dance quality metrics (style alignment, beat alignment, movement realism, choreography complexity) and video quality metrics (imaging quality, overall consistency).
Generalization: The method generalizes to unseen music tracks (including synthetic K-pop tracks not present in the training distribution), longer video sequences (up to 9 seconds, multiple times the training length), and unconventional subjects (e.g., animals).
Control: The system supports text-based control over dance style, setting, attire, and the number of dancers while maintaining music synchronization.
Responsiveness: The model responds dynamically to musical features; accelerating or decelerating the audio input results in corresponding changes in dance tempo and dynamics.
Ablation Studies: Results confirm that the proposed layer selection strategy, Beta-Uniform scheduling, and ZICA initialization are critical. Naive feature addition (similar to ControlNet) and non-zero initialization of cross-attention layers result in inferior performance.

Significance and Claims

The paper claims that MusicInfuser successfully bridges the gap between music and video generation for dance without intermediate motion data. By leveraging the rich choreographic knowledge already embedded in pre-trained text-to-video models, the approach generates novel, diverse, and high-quality dance movements that respond naturally to musical rhythms.

The authors emphasize that their method offers a flexible alternative to skeleton-based approaches, capturing visual nuances (hair, clothing, body dynamics) that skeletal representations miss. They assert that the framework generalizes well to unseen music and subjects, outperforming baselines in consistency and synchronization, all while being computationally efficient enough to train on a single GPU in a day. The work is presented as a step toward making video diffusion models "listen and dance," enabling controllable, music-driven video synthesis.