💻 computer science

MOVA: Towards Scalable and Synchronized Video-Audio Generation

MOVA is an open-source, 32B-parameter Mixture-of-Experts model designed for scalable and synchronized image-text-to-video-audio generation, providing high-quality, multimodal content including lip-synced speech and environment-aware sound effects.

Original authors: OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, Wenming Tu, Xiangyu Peng, Yang Gao, Yanru Huo, Ying Zhu, Yinze Luo, Yiyang Zh

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: OpenMOSS Team, Donghua Yu, Mingshu Chen, Qi Chen, Qi Luo, Qianyi Wu, Qinyuan Cheng, Ruixiao Li, Tianyi Liang, Wenbo Zhang, Wenming Tu, Xiangyu Peng, Yang Gao, Yanru Huo, Ying Zhu, Yinze Luo, Yiyang Zhang, Yuerong Song, Zhe Xu, Zhiyu Zhang, Chenchen Yang, Cheng Chang, Chushu Zhou, Hanfu Chen, Hongnan Ma, Jiaxi Li, Jingqi Tong, Junxi Liu, Ke Chen, Shimin Li, Shiqi Jiang, Songlin Wang, Wei Jiang, Zhaoye Fei, Zhiyuan Ning, Chunguo Li, Chenhui Li, Ziwei He, Zengfeng Huang, Xie Chen, Xipeng Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie where the actors’ lips move perfectly to the words, the sound of a glass breaking happens exactly when it hits the floor, and the background music swells perfectly with the mood of the scene.

Currently, most AI models struggle with this. They are like a talented painter who can draw a beautiful scene but is completely deaf, or a musician who can play a beautiful song but is blind. To make a "movie," they usually have to use two different tools: one to make the video and another to "glue" the sound on afterward. This often leads to a "bad dubbing" effect where the sound and picture feel slightly out of sync.

MOVA is like a creator who has finally learned to use both eyes and ears at the exact same time.

Here is a breakdown of how they did it, using some simple analogies:

1. The "Two-Brain" Architecture (The Dual-Tower)

Instead of one giant, confused brain trying to do everything, the researchers built MOVA with two specialized "brains" (called Towers) that talk to each other constantly:

The Video Brain: A massive expert at understanding shapes, light, and movement.
The Audio Brain: A specialist that understands rhythm, pitch, and tone.
The Bridge: This is the most important part. Imagine two dancers performing a duet. If they don't look at each other, they’ll trip. The "Bridge" acts like a constant stream of eye contact and physical touch, ensuring that when the Video Brain decides to move a mouth, the Audio Brain immediately knows to play the corresponding sound.

2. The "Perfect Script" (Data Engineering)

To teach this model, you can't just show it random videos. You have to give it a "perfect script."
If you show a child a video of a dog barking, you don't just say "dog." You say, "A golden retriever barks loudly in a sunny park while birds chirp in the background."
The MOVA team built a massive "automated teacher" (a pipeline) that looks at thousands of hours of video and writes incredibly detailed descriptions for both the sights and the sounds. This ensures the model learns exactly which sound belongs to which visual movement.

3. The "Smart Volume Knob" (Dual Classifier-Free Guidance)

When the AI is generating a video, it’s trying to follow two different bosses: the Text Prompt (what you asked for) and the Cross-Modal Connection (making sure the sound matches the video).
Sometimes, the AI might focus too much on the text and forget to sync the sound, or focus so much on the sync that the audio sounds robotic.
MOVA introduces a "smart knob" that allows users to balance these two. You can turn up the "Sync Knob" if you want perfect lip-syncing for a talking character, or turn up the "Creativity Knob" if you want a more cinematic, artistic video.

4. Why does this matter?

Before MOVA, high-quality video-audio generation was a "secret recipe" owned by big, closed companies (like OpenAI's Sora). MOVA is Open Source, which means they are handing the recipe to the entire world.

In short: MOVA isn't just a video maker or a sound maker; it is a multimedia storyteller that understands that in the real world, seeing and hearing are two sides of the same coin.

Technical Summary: MOVA (MOSS Video and Audio)

MOVA is an open-source, large-scale foundation model designed for the simultaneous and synchronized generation of high-quality video and audio. Developed by the SII-OpenMOSS Team, it addresses the limitations of current "cascaded" pipelines (where video and audio are generated separately) by employing an end-to-end joint multimodal modeling approach.

1. The Problem: The "Cascaded" Limitation

Existing video-audio generation systems typically rely on a two-step process: generating a video first and then synthesizing audio to match it (or vice versa). This approach suffers from several drawbacks:

Error Accumulation: Errors in the video generation stage propagate to the audio stage.
Lack of Interaction: The two modalities do not "talk" to each other during synthesis, leading to poor synchronization (e.g., unnatural lip-sync or environmental sounds that don't match visual physics).
Closed-Source Dominance: State-of-the-art models like Sora 2 and Veo 3 are proprietary, limiting research and community innovation in joint audio-visual modeling.

2. Methodology: Asymmetric Dual-Tower Architecture

MOVA utilizes a massive 32B parameter Mixture-of-Experts (MoE) architecture, with 18B parameters active during inference. Its technical core consists of:

Asymmetric Dual-Tower Design: Instead of training from scratch, MOVA couples a powerful pre-trained 14B Video Diffusion Transformer (DiT) with a 1.3B Audio DiT.
Bidirectional Bridge Module: A 2.6B parameter module connects the two towers. It uses bidirectional cross-attention to allow video hidden states to inform audio generation and vice versa, ensuring mutual integration of information.
Aligned RoPE (Rotary Positional Embedding): To solve the problem of different temporal densities (audio is much denser than video), the team modified RoPE to map both modalities onto a unified temporal scale, preventing "audio-visual drift."
Dual Sigma Shift (Decoupled Timesteps): Recognizing that audio and video have different noise sensitivities, the model samples timesteps independently for each modality. This allows for aggressive denoising for video while maintaining smoother, high-fidelity transitions for audio.
Dual Classifier-Free Guidance (Dual CFG): During inference, the model can independently scale guidance for the text prompt ( $s_T$ ) and the cross-modal bridge ( $s_B$ ). This provides a "knob" to balance semantic adherence (following the prompt) with synchronization precision (matching the audio to the video).

3. Data Engineering: Fine-Grained Annotation

The authors developed a sophisticated three-stage pipeline to curate high-quality data:

Preprocessing: Standardizing resolution (720p), frame rate (24fps), and segmenting clips using Voice Activity Detection (VAD) and scene transition analysis.
Quality Assessment: Filtering clips based on technical video quality (DOVER), audio aesthetics (Audiobox), and temporal/semantic alignment (SynchFormer/ImageBind).
Audio-Visual Captioning: Using MLLMs (like Qwen3-Omni and MiMo-VL) to generate modality-specific descriptions, which are then merged by a large LLM (GPT-OSS-120B) into a single, coherent, and semantically rich caption.

4. Key Contributions

Scalable Architecture: Proved that joint audio-video models can scale effectively with larger datasets and model parameters.
High-Fidelity Synchronization: Achieved state-of-the-art results in multi-speaker lip-syncing (in both English and Chinese) and environment-aware sound effects.
Open-Source Release: By releasing model weights, code, and a comprehensive training/inference stack, they provide a foundation for the research community.
Prompt Enhancement Workflow: Introduced a multi-stage pipeline that uses LLMs to "rewrite" user prompts, bridging the gap between simple user input and the complex descriptions the model was trained on.

5. Results and Evaluation

Quantitative Superiority: MOVA outperforms baselines (like LTX-2 and Ovi) in several metrics, including Inception Score (IS) for audio diversity, DNSMOS for speech quality, and DeSync for temporal alignment.
Lip-Sync Precision: It demonstrates superior performance in LSE-C (Confidence) and LSE-D (Distance) metrics, indicating highly accurate mouth movements.
Human Preference: In an "Arena-style" human evaluation, MOVA achieved a significantly higher ELO rating (1113.8) compared to existing open-source models, proving its superior perceptual quality.
Emergent T2VA Capability: The model shows it can generate synchronized content from text alone (Text-to-Video-Audio) even when provided with a blank image placeholder.

6. Significance

MOVA represents a major step toward truly multimodal generative AI. By moving away from cascaded pipelines toward a unified, bidirectional architecture, it sets a new benchmark for how machines can understand and generate the complex, intertwined relationship between what we see and what we hear. Its open-source nature is particularly significant for democratizing high-end video-audio production tools.