💻 computer science

Native Audio-Visual Alignment for Generation

NAVA is a novel framework for joint audio-video generation that employs a native audio-visual alignment strategy within an Align-then-Fuse MMDiT architecture to achieve superior synchronization, video quality, and controllable speech timbre while using only 6.3B parameters.

Original authors: Longbin Ji, Guan Wang, Xuan Wei, Chenye Yang, Xiangrui Liu, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Jingzhou He

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Longbin Ji, Guan Wang, Xuan Wei, Chenye Yang, Xiangrui Liu, Zhenyu Zhang, Shuohuan Wang, Yu Sun, Jingzhou He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to create a movie scene where a character is riding a bicycle while talking. You need two things to happen perfectly together: the video of the bike moving and the sound of the wind and the person's voice.

For a long time, computers tried to make these two things separately and then glued them together at the end. This is like having one artist draw the bike and a completely different artist record the voice, then trying to make them match up later. Often, the voice would be slightly out of sync with the lip movements, or the sound wouldn't feel like it belonged in that specific scene.

Other newer methods tried to put the artist, the voice actor, and the director all in the same tiny room to work on everything at once. While this helps them talk to each other, it gets messy. The "director" (the text instructions) ends up getting confused with the "sound engineer" (the timing of the noise), making it hard to control specific details like who is speaking or what their voice sounds like.

Enter NAVA: The "Rehearsal Room" Approach

The paper introduces NAVA (Native Audio-Visual Alignment), which changes the workflow. Instead of working separately or getting everything mixed up in one big pot, NAVA uses a clever two-step process called "Align-then-Fuse."

Think of it like a theater rehearsal:

The Rehearsal (Alignment): First, the actors (audio) and the stage crew (video) meet in a dedicated "rehearsal room." Here, they practice together without the director giving them new lines. They figure out exactly how the sound of a footstep matches the visual of a foot hitting the ground, or how a guitar strum matches a hand movement. They build a natural, native connection between sound and sight.
The Performance (Fusion): Once they know how to move and speak together, the director (the text prompt) steps in. The director doesn't need to teach them how to sync up anymore; they just tell them what to do. "Now, the character should say this line with a grumpy voice." Because the actors and crew already know how to move in sync, they can instantly adapt to the new instructions without losing their timing.

The Secret Sauce: "Timbre-in-Context"

NAVA also has a special trick for handling multiple speakers. Imagine a script where a mother and her child are talking. You don't want the mother's voice to sound like the child's.

Older methods might have a separate "voice switch" button that changes the whole scene's voice. NAVA is smarter. It treats the voice style (timbre) as part of the script itself. It attaches a specific "voice tag" to the mother's lines and a different "voice tag" to the child's lines. When the computer generates the scene, it knows exactly which voice belongs to which sentence, just like a director reading a script with notes on who is speaking.

What the Paper Found

The researchers tested NAVA against other top models using a benchmark called Verse-Bench (a test of how well audio and video match) and Seed-TTS (a test of voice control).

Better Sync: NAVA was the best at making sure the sound happened at the exact moment the visual event occurred (like a door slamming or lips moving).
High Quality: The video looked great, and the audio sounded clear and realistic.
Efficiency: Despite being smaller than many competitors (using only 6.3 billion "brain cells" or parameters), it outperformed much larger models.
Control: It could switch between different speakers in the same scene seamlessly, keeping the voices distinct and accurate.

In Summary

NAVA solves the problem of making sound and video by giving them a dedicated space to learn how to move together before they try to follow complex instructions. It's like teaching a dance duo to master their steps before the choreographer tells them which song to dance to. The result is a movie scene where the sound and picture feel like they were born together, not just glued together.

Technical Summary: NAVA - Native Audio-Visual Alignment for Generation

Problem Statement

Joint audio-video generation aims to synthesize temporally synchronized and semantically coherent visual-acoustic content. While commercial systems (e.g., Seedance, Kling, Veo) have demonstrated potential, their architectures remain proprietary. Open-source research has largely followed two paradigms, both of which exhibit significant limitations:

Dual-Tower Designs: These methods generate audio and video in separate streams, introducing cross-modal interaction only through late-stage alignment modules. This "posterior alignment" weakens the fine-grained co-evolution of audio and video, making synchronization dependent on auxiliary modules rather than a unified generative representation.
Fully Unified Tri-Modal Designs: These approaches mix textual context, audio, and video tokens into a single shared attention space. While enabling direct interaction, this design couples high-level semantic conditioning with low-level audio-visual synchronization. Consequently, semantic guidance and temporal alignment are optimized in the same space, potentially hindering the formation of a dedicated synchronization structure.

The core challenge is to achieve precise event-level correspondence and temporal consistency without sacrificing semantic controllability or requiring massive parameter counts.

Methodology: NAVA Framework

The authors propose NAVA (Native Audio-Visual Alignment), a framework that decouples audio-visual synchronization from external context conditioning. The core philosophy is to first establish audio-video correspondence in a dedicated interaction space and subsequently use external context to condition the joint denoising process.

1. Formulation: Context-Conditioned Native Alignment

Unlike dual-tower methods (which align after separate generation) or fully unified methods (which mix all tokens), NAVA separates the roles of synchronization and conditioning:

Native Synchronization: Audio ( $h_a$ ) and video ( $h_v$ ) tokens interact via self-attention in a dedicated space without context tokens: $[h'_a, h'_v] = \text{SelfAttn}([h_a, h_v])$ . This allows acoustic patterns and visual dynamics to co-evolve.
External Conditioning: Context tokens ( $c$ ), including text and timbre cues, are injected via cross-attention after the initial alignment: $[\tilde{h}_a, \tilde{h}_v] = \text{CrossAttn}([h'_a, h'_v], c)$ .

2. Architecture: Align-then-Fuse MMDiT

NAVA is instantiated using a modified Multi-modal Diffusion Transformer (MMDiT) with a progressive design:

Hierarchical Alignment Layers (Early): The first 10 blocks use Modality-Decoupled Alignment Projections. Since audio and video differ in spatiotemporal structure and token rates, these layers map heterogeneous tokens into a shared interaction space using modality-specific projections before applying joint self-attention. This stabilizes early-stage correspondence learning.
- Token Rate Handling: Audio rotary positional embeddings are rescaled ( $\theta_{rope} = TR_v / TR_a$ ) to align temporal coordinates with video tokens.
Unified Fusion Layers (Late): The remaining 20 blocks use Modality-Shared Unified Projections. Once correspondence is established, audio and video tokens share generation parameters to encourage compact collaborative denoising. Context remains external via cross-attention to modulate the trajectory without disrupting the learned synchronization.

3. Timbre-in-Context Conditioning

To achieve controllable speech timbre (e.g., multi-speaker dialogue), NAVA introduces a mechanism that binds reference timbre cues to specific speech spans within the prompt structure.

Reference timbre embeddings are extracted and treated as context tokens.
These tokens are inserted directly into the textual prompt sequence corresponding to specific speech segments (e.g., $\langle S \rangle, s_i, \text{Text}(S_i), \langle E \rangle$ ).
This allows flexible content-timbre binding without auxiliary speaker-control branches or backbone modifications.

4. Training and Inference

Progressive Multi-Task Training: The model is trained in three stages: (1) Audio-only and paired data to initialize audio pathways; (2) High-quality audio and full audio-visual data to improve fidelity and sync; (3) Fine-tuning on curated data for instruction following and complex control.
Structured Dropout: Random cross-modality attention masking and timbre token dropping are used to create paired conditional/unconditional paths for guidance.
Condition-Factorized Classifier-Free Guidance (CFG): During inference, three distinct guidance directions are computed and combined:
- $\Delta_{text}$ : Text adherence.
- $\Delta_{align}$ : Audio-visual synchronization (by masking audio-video cross-attention).
- $\Delta_{timbre}$ : Timbre preservation (by dropping timbre tokens).
  This allows independent control over synchronization and timbre without retraining.

Key Contributions

NAVA Framework: A Native Audio-Visual Alignment approach that formulates joint generation as context-conditioned alignment, enabling precise event-level correspondence modeling with pretrained video backbones.
Align-then-Fuse MMDiT: An architecture that transitions from modality-aware alignment to modality-shared denoising, balancing synchronization stability with collaborative generation efficiency.
Timbre-in-Context Conditioning: A novel mechanism for segment-level timbre control that integrates reference cues directly into the context pathway, supporting multi-speaker generation without auxiliary branches.
Performance: Extensive experiments demonstrate that NAVA achieves superior synchronization and video quality with significantly fewer parameters (6.3B) compared to larger baselines.

Experimental Results

Quantitative Evaluation (Verse-Bench)

NAVA (6.3B parameters) outperforms or competes with significantly larger models (10B–32B):

Synchronization: Achieved the highest Sync-C (7.791) and lowest Sync-D (7.566), indicating superior lip-audio alignment.
Video Quality: Scored 0.659 (IB), the best among compared models.
Audio Quality: Achieved the lowest Word Error Rate (WER) of 0.099 and competitive Production Quality (PQ) and Fréchet Distance (FD).
Efficiency: NAVA uses the fewest parameters (6.3B) among all compared audio-video models while delivering top-tier performance.

Timbre Controllability (Seed-TTS)

On the Seed-TTS benchmark, NAVA achieved a speaker similarity of 66.7% and a WER of 4.20.

It substantially outperformed the audio-video baseline DreamID-Omni (35.7% similarity, 31.76 WER).
It remained competitive with pure audio-only models (e.g., CosyVoice2: 65.2% similarity), demonstrating that joint generation does not inherently degrade timbre control.

Qualitative and User Study

Visualizations: NAVA successfully generated synchronized content in complex scenarios, including dynamic motion, musical performance, multi-speaker dialogue, and shot transitions.
User Study (GSB Protocol): In pairwise comparisons against Ovi-1.1, LTX-2.3, and daVinci:
- T2AV: NAVA won 67.5% (vs Ovi), 60.0% (vs LTX), and 80.0% (vs daVinci) in overall quality. It won even more decisively in audio-visual alignment (62.5%–72.5%).
- TI2AV: NAVA showed clear advantages in alignment against most baselines and remained competitive in overall quality.

Significance and Claims

The paper claims that NAVA represents a significant step forward by decoupling the synchronization mechanism from semantic conditioning. By establishing a dedicated alignment space before applying context, the model avoids the trade-offs inherent in dual-tower (weak co-evolution) and fully unified (entangled control) paradigms.

The authors assert that this "Native Audio-Visual Alignment" approach enables:

Precise Event-Level Correspondence: Modeling speech-lip motion and scene-dependent acoustics directly within the generation trajectory.
Scalability and Efficiency: Achieving state-of-the-art results with a compact 6.3B parameter model.
Flexible Control: Enabling compositional multi-speaker generation through context-based timbre binding.

The authors acknowledge limitations in generating long-tail events (rare animal sounds, complex music mixtures) and suggest that future work could explore deeper audio-visual coupling via joint tokenizers or unified representation models. However, they conclude that native alignment with decoupled context conditioning is a promising direction for scalable and controllable audio-video generation.