Technical Summary: MACE-Dance

Problem Statement

The task of music-driven dance video generation aims to synthesize high-quality dance videos from audio inputs and a reference image. Despite advances in related fields such as music-driven 3D dance generation, pose-driven image animation, and audio-driven talking-head synthesis, existing approaches fail to adequately address this specific task due to fundamental mismatches in generation targets and constraints.

• 3D Dance Generation: While effective at producing kinematically plausible motions, these methods often lack realistic human-scene interactions and high-fidelity visual textures when rendered into 2D video.
• Pose-Driven Image Animation: These methods rely on manual pose design, which is time-consuming and difficult for complex full-body dance sequences. Furthermore, existing audio-driven methods (e.g., talking heads) focus on simple upper-body gestures rather than the complex full-body locomotion required for dance.
• Direct End-to-End Approaches: Current limited research on direct music-to-video generation often fails to capture the inherently 3D nature of dance, resulting in compromised motion quality (e.g., physical implausibility) and visual appearance (e.g., blurring, artifacts).

The core challenge lies in simultaneously achieving (1) kinematically plausible and artistically expressive dance motions, and (2) high-fidelity visual appearance with strong spatiotemporal consistency.

Methodology: MACE-Dance

The authors propose MACE-Dance, a framework utilizing a Cascaded Mixture-of-Experts (MoE) architecture. The system decouples the complex task into two specialized stages: a Motion Expert and an Appearance Expert, connected by an explicit 3D motion representation (SMPL parameters) rather than 2D keypoints. This choice preserves full-body geometric structure, disentangles pose from viewpoint, and provides robustness against occlusion.

1. Motion Expert (Music-to-3D Motion)

This module generates 3D motion sequences conditioned on music, enforcing kinematic plausibility and artistic expressiveness.

• Architecture: A Diffusion Model with a BiMamba-Transformer hybrid backbone.
  • BiMamba: Captures intra-modal local dependencies in music and dance sequences using bidirectional state-space models, ensuring fine-grained temporal continuity.
  • Transformer: Models cross-modal global context, aligning the overall dance structure with long-term musical phrasing via cross-attention.
  • Non-Autoregressive Generation: The model generates the entire sequence in one pass, improving efficiency and avoiding exposure bias common in autoregressive methods.
• Training Strategy: The authors employ Guidance-Free Training (GFT) instead of conventional Classifier-Free Guidance (CFG). GFT trains a single model to implicitly represent temperature-controlled sampling behavior, mitigating distribution mismatch and enhancing generation stability.
• Loss Functions: The model is optimized using a weighted sum of reconstruction loss, 3D joint loss, velocity loss, and foot contact loss to ensure physical realism.

2. Appearance Expert (Motion-and-Reference Conditioned Video Synthesis)

This module synthesizes the final video by animating a reference image using the generated 3D motion, preserving visual identity and spatiotemporal coherence.

• Base Model: Built upon Wan-Animate, a general-purpose character animation model.
• Decoupled Fine-Tuning Strategy: To address the domain gap between general animation and complex dance, the authors introduce a two-stage fine-tuning approach:
  • Kinematic Stage: Fine-tunes only the Body Adapter to strengthen kinematic conditioning and motion adherence, ensuring the generated video strictly follows the 3D motion guidance without altering the backbone's generative priors.
  • Aesthetic Stage: Attaches lightweight LoRA adapters to the attention and feed-forward projections of each DiT block. This stage refines visual quality, enhancing texture fidelity (skin, hair, fabric) and handling complex camera choreography while preserving the subject's identity.
• Projector: A 3D-to-2D Motion Projector converts the SMPL sequence from the Motion Expert into 2D keypoints required by the Appearance Expert, bridging the 3D and 2D domains.

Key Contributions

1. Dataset and Benchmark: The authors curated MA-Data, a large-scale dataset comprising 70k clips (116 hours) spanning over 20 dance genres. It combines 3D-rendered data (motion-centric) and in-the-wild internet data (appearance-centric). They also designed a comprehensive motion–appearance evaluation protocol covering kinematic fidelity, diversity, synchronization, and visual quality metrics.
2. MACE-Dance Framework: A cascaded MoE framework that achieves State-of-the-Art (SOTA) performance in music-driven dance video generation by synergistically integrating motion and appearance generation.
3. Motion Expert Innovations: The introduction of a BiMamba-Transformer hybrid architecture and Guidance-Free Training (GFT) strategy, achieving SOTA performance in music-driven 3D dance generation on the FineDance dataset.
4. Appearance Expert Innovations: The development of a decoupled Kinematic–Aesthetic fine-tuning strategy, achieving SOTA performance in pose-driven image animation on the MA-Data dataset.

Results

Extensive experiments demonstrate that MACE-Dance outperforms existing baselines (including 3D generation methods pipelined with animation models, and general human-motion video generators) across multiple metrics:

• Motion Quality: On the FineDance dataset, the Motion Expert achieves the best FID (17.83), diversity (DIV), and Beat Alignment Score (BAS), indicating high fidelity, diversity, and music synchronization.
• Video Quality: On the MA-Data dataset, MACE-Dance achieves SOTA scores in imaging quality (IQ), aesthetic quality (AQ), subject consistency (SC), and temporal flickering (TF).
• User Study: A double-blind user study with 40 participants confirmed that MACE-Dance is overwhelmingly preferred over baselines (e.g., Hallo2, EDGE, Lodge, MEGA) across all six dimensions: Dance Synchronization, Dance Quality, Dance Creativity, Perceptual Quality, Temporal Consistency, and Identity Consistency.
• Long-Sequence Generation: The framework successfully generates coherent long-sequence videos (up to 34 seconds in inference) without significant motion drift or visual degradation, leveraging the BiMamba-Transformer's length extrapolation and pose-anchored relay rendering.

Significance and Claims

The paper claims that MACE-Dance represents a significant advancement in the field of AIGC for dance. By explicitly decoupling the task into motion and appearance generation and utilizing a 3D intermediate representation, the framework overcomes the limitations of end-to-end models that struggle with spurious cross-modal correlations and physical plausibility.

The authors emphasize that their approach not only sets a new benchmark for video quality and motion expressiveness but also provides a controllable and interpretable pipeline. The explicit 3D motion output allows for downstream applications such as motion editing, retargeting to different character rigs, and potential use in VR/AR and embodied AI, distinguishing it from black-box video generation models. The work underscores the importance of specialized expert modules and structured intermediate representations in solving complex, multi-modal generation tasks.