🤖 AI

OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation

OmniNFT is a novel modality-aware online diffusion reinforcement learning framework that enhances joint audio-video generation by addressing multi-objective inconsistency, gradient imbalance, and uniform credit assignment through modality-wise advantage routing, layer-wise gradient surgery, and region-wise loss reweighting.

Original authors: Guohui Zhang, XiaoXiao Ma, Jie Huang, Hang Xu, Hu Yu, Siming Fu, Yuming Li, Zeyue Xue, Lin Song, Haoyang Huang, Nan Duan, Feng Zhao

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guohui Zhang, XiaoXiao Ma, Jie Huang, Hang Xu, Hu Yu, Siming Fu, Yuming Li, Zeyue Xue, Lin Song, Haoyang Huang, Nan Duan, Feng Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to create a movie where the sound and the picture are perfectly in sync. You want the robot to learn from its mistakes, just like a human does. This is what the paper calls "Reinforcement Learning."

However, the authors found that when they tried to teach the robot to make both audio and video at the same time using standard methods, the robot got confused. It was like trying to coach a basketball player and a singer simultaneously, but the coach was shouting contradictory instructions.

Here is the story of OmniNFT, the new method the authors built to fix this, explained through simple analogies.

The Three Big Problems

The authors discovered three specific reasons why the robot was failing:

The "Confused Coach" Problem (Advantage Inconsistency):
Imagine a coach grading a student's performance. Sometimes, the student's video is amazing, but the audio is terrible. In standard training, the coach might give a single "average" grade for the whole performance.
- The Issue: If the video is great but the audio is bad, an average grade might tell the robot, "You did okay," which doesn't help it fix the audio. Or worse, the robot might think it needs to change the video to fix the audio, making the video worse.
- The Fix: OmniNFT acts like a coach with two separate scorecards. One scorecard judges the video, and the other judges the audio. If the video is good, the video part of the robot gets a "high five." If the audio is bad, the audio part gets a "time out." They don't mix the scores up.
The "Noisy Classroom" Problem (Gradient Imbalance):
Think of the robot's brain as a multi-story building.
- The bottom floors (shallow layers) are where the audio is generated (making the voice sound right).
- The top floors (deep layers) are where the video and audio talk to each other to stay in sync.
- The Issue: When the robot tries to learn from the video, the "noise" from the video lessons was accidentally leaking down into the bottom floors, messing up the audio generation. It was like the video teacher shouting instructions in the audio classroom, confusing the audio students.
- The Fix: The authors installed a soundproof wall (Gradient Surgery). They let the video and audio talk to each other on the top floors where they need to sync up, but they blocked the video noise from reaching the bottom audio floors. This lets the audio learn to be clear without being distracted by the video.
The "Spotlight" Problem (Uniform Credit Assignment):
Imagine a scene where a character is speaking. The most important part of the video is their mouth moving, and the most important part of the audio is their voice.
- The Issue: Standard training treats every pixel and every sound wave equally. It's like shining a giant, flat floodlight over the whole stage. The robot spends just as much time learning about the background wall as it does learning about the actor's lips.
- The Fix: OmniNFT uses a smart spotlight. It looks at the video and sees exactly where the sound is coming from (like a person's mouth). It then shines a bright, intense light on just those specific areas. This tells the robot, "Pay extra attention here! This is where the magic happens."

The Result: A Better Movie Maker

By using these three tricks, the authors tested their new system (called OmniNFT) on a powerful existing model called LTX-2.

The results were impressive:

Better Quality: The videos looked sharper, and the sounds were clearer.
Better Sync: The lips moved exactly when the words were spoken.
Better Harmony: The video and audio felt like they belonged together, rather than two separate things glued together.

In a Nutshell

The paper says that to make great AI movies, you can't just use a "one-size-fits-all" teaching method. You have to:

Grade the sound and picture separately.
Stop the video lessons from confusing the audio lessons.
Focus extra attention on the parts of the movie where sound and picture actually meet (like a talking face).

OmniNFT does exactly this, resulting in AI-generated videos that look and sound much more realistic and synchronized than before.

Technical Summary: OmniNFT

Problem Statement

Recent advances in joint audio-video generation have yielded promising results, yet real-world applications demand a combination of high per-modality fidelity, robust cross-modal semantic consistency, and fine-grained audio-video synchronization. While Reinforcement Learning with Verifiable Rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), has shown success in text-to-image/video generation, its direct application to joint audio-video generation remains suboptimal.

The authors identify three primary optimization bottlenecks that hinder the effective application of vanilla RL to this multi-objective, multi-modal task:

Multi-objective Advantages Inconsistency: The rewards for video and audio components within a single multimodal output are often inconsistent. High-quality video does not necessarily correlate with high-quality audio, leading to conflicting gradient signals when a single global advantage is used.
Multi-modal Gradients Imbalance: In dual-stream diffusion models, intra-modal generation primarily occurs in shallow layers, while cross-modal interaction happens in deeper layers. However, gradients from the video branch tend to dominate the update direction of shallow audio layers during backpropagation, disrupting the intra-modal audio generation process.
Uniform Credit Assignment: Audio-video synchronization and fine-grained alignment rely on specific critical regions (e.g., sound-emitting areas). Standard RL updates treat all regions uniformly, failing to prioritize these crucial areas for exploration and optimization.

Methodology: OmniNFT

To address these challenges, the authors propose OmniNFT (Modality-wise Omni Diffusion Negative-aware Fine-Tuning), a novel modality-aware online diffusion RL framework. The method operates on a dual-stream diffusion backbone (specifically LTX-2) and introduces three coordinated innovations:

1. Modality-wise Advantage Routing

Instead of computing a single scalar advantage from all rewards, OmniNFT computes independent advantages for each reward function (video quality, audio quality, and audio-video synchronization).

Routing Logic: The video advantage ( $A_v$ ) and audio advantage ( $A_a$ ) are routed exclusively to their respective generation branches. The synchronization advantage ( $A_{av}$ ), which measures alignment, is broadcast as shared supervision to both branches.
Composite Advantage: The routed advantages are combined as $\tilde{A}_v = A_v + A_{av}$ and $\tilde{A}_a = A_a + A_{av}$ . This design ensures modality-specific supervision for uni-modal rewards while preserving shared cross-modal signals, mitigating reward inconsistency.

2. Layer-wise Gradient Surgery

To prevent video gradients from corrupting shallow audio layers responsible for intra-modal generation, the authors implement a selective gradient detachment mechanism.

Mechanism: In the Audio-to-Video (A2V) cross-attention blocks, the Key ( $K$ ) and Value ( $V$ ) vectors from the audio stream are partially detached in the shallow layers.
Implementation: A stop-gradient operator is applied to a portion of the audio KV pairs in layers $l < L$ (where $L$ is a shallow boundary, e.g., 10). The gradient flow is scaled by $(1 - \alpha)$ , where $\alpha$ is a detach ratio. This allows RL gradients to flow freely through deep layers (responsible for cross-modal alignment) while suppressing leakage into shallow audio layers.

3. Region-wise Loss Reweighting

To address uniform credit assignment, the framework identifies and prioritizes critical regions within the video frames that are semantically linked to audio.

Intrinsic Proxy: The authors utilize the Video-to-Audio (V2A) cross-attention maps from the audio branch as an intrinsic indicator of sound-emitting regions. These maps naturally highlight speaking subjects and relevant visual areas.
Weighting Strategy: Attention maps from deep blocks and late denoising steps are aggregated to generate a per-token score. These scores are normalized and used to reweight the video branch loss, concentrating optimization capacity on perceptually critical regions while leaving the audio loss standard.

Key Contributions

Problem Identification: The paper provides an in-depth analysis revealing that vanilla RL fails in joint audio-video generation due to advantage inconsistency, gradient imbalance, and uniform credit assignment.
Framework Proposal: OmniNFT is introduced as a solution combining modality-wise advantage routing, layer-wise gradient surgery, and region-wise loss reweighting to facilitate multi-objective and multi-modal optimization.
Empirical Validation: Extensive experiments demonstrate that OmniNFT achieves consistent gains over strong baselines in perceptual quality, cross-modal alignment, and synchronization without introducing significant computational overhead.

Experimental Results

The authors evaluated OmniNFT using the LTX-2 backbone on JavisBench and VBench.

Perceptual Quality: OmniNFT improved Visual Quality (VQ) from 2.038 to 3.326 (+63.2%) and Audio Quality (AQ) from 5.197 to 5.715 (+10.0%) compared to the base LTX-2 model. It also showed notable gains in VBench imaging quality (+10.5%).
Cross-Modal Consistency: The method improved Text-Audio ImageBind (TA-IB) by 15.2% over LTX-2 and outperformed the larger LTX-2.3 model. The CLAP score (audio-text alignment) increased from 0.412 to 0.445.
Synchronization: A significant reduction in DeSync (a measure of desynchronization) was observed, dropping from 0.569 to 0.269 (-52.7%), substantially outperforming both the base model and the GDPO baseline (0.412).
Ablation Studies: The study confirmed that each component contributes uniquely: advantage routing improved consistency and synchrony; gradient surgery enhanced audio fidelity; and region-wise reweighting pushed AV-consistency and synchrony to their peak performance.

Significance and Claims

The paper claims that OmniNFT represents a significant step forward in aligning generative models with the complex, multifaceted requirements of real-world audio-video applications. By explicitly addressing the optimization mismatches inherent in multi-modal RL, the framework enables simultaneous improvement in fidelity, alignment, and synchronization. The authors position this work as the first to systematically identify and resolve these specific bottlenecks in joint audio-video generation, offering a scalable and effective post-training paradigm that outperforms existing RLVR approaches and larger baseline models.