💻 computer science

OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films

OmniVR is a pioneering 22B-parameter joint audio-video generative model that restores degraded historical films by formulating the task as conditional generation within a unified multimodal DiT, simultaneously addressing visual and audio degradations while ensuring cross-modal consistency and natural colorization.

Original authors: Xin Lu, Zihao Fan, Mingchen Zhong, Jie Huang, Xueyang Fu, Zheng-Jun Zha

Published 2026-08-06

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xin Lu, Zihao Fan, Mingchen Zhong, Jie Huang, Xueyang Fu, Zheng-Jun Zha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a time traveler with a magical camera that can only see the past. You find an old, dusty film reel from a century ago. When you play it, the picture is fuzzy, flickering, and black-and-white, while the sound is a crackly, hissing mess that sounds like it's coming from inside a tin can. For decades, scientists have tried to fix these movies, but they usually treat the picture and the sound as two separate problems. They have a team of artists to clean up the video and a different team of audio engineers to fix the noise, working in isolation. The problem is, in the real world, the video and audio are best friends; they influence each other. If the audio is muffled, it's hard to tell if a character is whispering or just far away. If the video is blurry, it's hard to tell if a sound effect matches the action.

This paper introduces a new kind of "digital restorer" called OmniVR. Think of it not as two separate repair crews, but as a single, super-smart detective who looks at the messy picture and the noisy sound together to figure out what the original scene really looked and sounded like. The researchers built this detective using a massive, 22-billion-parameter brain (a type of AI model) that was originally taught to create new movies from scratch. Instead of just making things up, they taught this brain to "un-diffuse" the mess, using the bad parts of the old film as clues to reconstruct the clean parts. They didn't just fix the blur or the hiss; they made the video and audio talk to each other, ensuring that when a character moves their lips, the sound matches perfectly, and when the music swells, the scene feels right.

The Big Problem: Why Fixing One Side Isn't Enough

Historical films are like time capsules, but they are often broken. The paper points out that these old movies suffer from a double whammy: the video gets blurry, grainy, and flickers, while the audio gets hissy, clipped, and drops out. The authors noticed something crucial that previous methods missed: the damage happens at the same time. You can't just fix the picture and hope the sound magically gets better, or vice versa. In fact, if you fix the video but leave the audio broken (or the other way around), the two stop syncing up. It's like trying to dance with a partner who is moving to a different beat; the result feels awkward and unnatural.

The researchers found that in the vast majority of old clips they studied, both the eyes and the ears were suffering together. They also discovered that the brain of an AI model designed to generate video and audio together actually has "cross-wiring." The audio part of the model pays attention to the video, and the video part pays attention to the audio. If you silence the audio, the video generation changes. This means that to truly restore a film, you have to let the audio and video help each other, rather than treating them as strangers.

The Solution: A Team of Three

To build OmniVR, the authors created a three-step strategy to turn a "movie-maker" AI into a "movie-restorer" AI.

1. The "Fake Old Movie" Factory
First, the team needed to teach the AI what a broken old movie looks like. Since they couldn't find enough real broken movies with perfect "before and after" copies to train on, they built a digital factory. They took high-quality, modern videos and audio and deliberately ruined them. They added digital scratches, dust, flicker, and blur to the video, and hiss, popping, and muffled sounds to the audio. They made sure these "fake" damages looked and sounded exactly like the real things found in history books. This allowed the AI to practice fixing problems it had never seen before, learning to spot the difference between a real face and a blurry mess.

2. The "Silent Partner" Trick
The AI they started with was a giant model designed to create movies from text descriptions (like "a cat running"). They didn't want to retrain the whole thing from scratch because that would take forever and might make it forget how to be creative. Instead, they used a clever trick. They told the AI: "Keep your ability to make movies, but now, instead of listening to a text description, listen to the broken movie we give you."
They did this by feeding the bad video and bad audio into the AI's "ears" (its input channels) while keeping the "brain" mostly the same. They also used a special technique called prompt annealing. Imagine the AI is learning a new language. At first, they let it read the original text descriptions of the scenes to keep it warm. Then, slowly, they replaced those descriptions with a single, fixed instruction: "Make this sharp, clear, and synchronized." This helped the AI transition smoothly from "making new things" to "fixing old things" without losing its creative spark.

3. The "Anchor" for Long Movies
Old movies can be very long, but the AI can only process short chunks at a time (about 5 seconds or 121 frames). If you just stitch these chunks together, the end of one chunk might look slightly different from the start of the next, causing the video to jump or the color to shift. To fix this, the authors used a "first-frame anchor." When the AI finishes fixing one chunk, it takes the very last frame it created and uses it as the "starting point" for the next chunk. It's like a relay race where the runner hands the baton to the next person; the next runner starts exactly where the last one left off. This keeps the movie smooth and continuous, even for long films.

What They Found: A New Standard for Restoration

The team tested OmniVR on a new benchmark they created called OmniVRBench, which includes 200 real historical clips. They compared their method against the best existing tools, which usually fix video and audio separately or just add color to black-and-white films.

The results were clear: OmniVR is the first method to successfully fix the video, the audio, and the color all at once.

Visual Quality: On a scale measuring how "natural" and sharp the image looks, OmniVR scored significantly higher than any other method. It didn't just remove the blur; it added back fine details like skin texture and fabric patterns that were lost.
Audio Quality: The restored sound was much clearer, with less hiss and better volume. It sounded more like a human speaking and less like a robot in a tin can.
Synchronization: Because the video and audio were fixed together, the lips moved perfectly with the words. Other methods often made the audio sound good but the lips looked out of sync, or vice versa.
Color: It didn't just add random colors; it added plausible colors that matched the lighting and mood of the scene, making black-and-white films look like they were shot in color.

The researchers also showed that simply combining a video fixer and an audio fixer (a "cascade" approach) didn't work as well. The separate tools couldn't talk to each other, so they often made mistakes that the joint model avoided. For example, a separate audio fixer might change the timing of the sound just enough to make it out of sync with the video, but OmniVR kept them locked together.

The Bottom Line

OmniVR proves that restoring history isn't just about cleaning up a picture or a sound file; it's about understanding the relationship between the two. By treating the video and audio as a single, interconnected story, the model can recover details that were previously thought lost. While the paper notes that extremely damaged films (where frames are missing entirely) can still be tricky, and that the AI might occasionally "hallucinate" a tiny bit of detail to fill in the gaps, the results are a massive leap forward. It offers a way to bring the past back to life, not just as a static image, but as a living, breathing, synchronized experience. The code and the model are being released to the public, meaning anyone can now use this "time-traveling detective" to restore their own old memories.

Technical Summary: OmniVR – Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films

1. Problem Statement

Historical films suffer from co-occurring degradations in both visual and acoustic streams, including blur, noise, flicker, compression artifacts, hiss, clipping, and bandwidth loss. Existing restoration methods typically treat these modalities independently: visual restoration models (e.g., DeepRemaster, RealBasicVSR) ignore the soundtrack, while audio restoration tools (e.g., VoiceFixer) operate without visual context. This separation leads to quality gaps and cross-modal inconsistencies, such as temporal misalignment between restored faces and unstable soundtracks, or flickering frames paired with cleaned audio. The paper argues that historical film restoration is fundamentally an audio-visual problem requiring a unified approach to preserve temporal alignment and mutual consistency between modalities.

2. Methodology

OmniVR is the first joint audio-video generative restoration model, built upon a 22B-parameter multimodal Diffusion Transformer (DiT) backbone (LTX-2). It formulates restoration as a conditional generation task where low-quality (LQ) video and audio serve as latent conditions to guide the denoising process. The methodology relies on three key technical designs:

A. Joint Audio-Video Degradation Pipeline

To train on clean internet-collected clips while simulating real archival characteristics, the authors employ an online degradation operator $\mathcal{D}$ . This pipeline synthesizes LQ pairs $(V_l, A_l)$ from high-quality pairs $(V_h, A_h)$ using independent random branches for visual and audio modalities.

Visual Branch: Extends a Real-ESRGAN-style high-order model with temporally coherent old-film artifacts (flicker, scratches, dust, abrasion bands, grayscale conversion).
Audio Branch: Combines legacy analog defects (hiss, rumble, clicks, clipping, bandwidth loss) with generative model failure modes (spectral over-smoothing, harmonic thinning).
Severity Scalars: Each sample is assigned normalized severity scalars ( $d_v, d_a$ ) to facilitate loss reweighting.

B. Architecture-Preserving T2AV-to-AV2AV Transition

The model adapts a pre-trained Text-to-Audio-Video (T2AV) foundation model for Audio-Video-to-Audio-Video (AV2AV) restoration without altering the backbone topology.

Channel-Concat Condition Injection: The degraded latent conditions are injected via channel concatenation with the noisy target tokens. New patchify parameters are initialized near zero, allowing the model to start as a functional T2AV generator and gradually learn to incorporate LQ conditions.
Prompt Annealing: The text condition transitions from diverse descriptive captions (early training) to a fixed restoration prompt ("sharp, stable, denoised video with clean synchronized audio") via linear annealing. This preserves the generative prior of the pre-trained model while shifting the task to restoration without requiring per-clip captions at inference.
Cross-Modal Interaction: Learnable cross-modal gates enable audio and video tokens to attend to each other, facilitating mutual cues (e.g., lip motion guiding audio, audio energy guiding visual sharpness).

C. First-Frame Anchoring and Optimization

To enable coherent long-video restoration and ensure audio fidelity:

First-Frame I2V Anchoring: The model operates on 121-frame windows. During training, the first frame of the window is occasionally replaced with its clean latent token (with probability $p_{ff}=0.5$ ), and the loss on these anchor tokens is masked. At inference, the restored last frame of one window serves as the first-frame anchor for the next, ensuring temporal continuity.
Loss Reweighting: A degradation-aware reweighting scheme adjusts the loss contribution of video and audio based on their respective severity scalars ( $d_v, d_a$ ), preventing the model from being dominated by the easier modality and prioritizing jointly degraded samples.
Waveform Supervision: An optional multi-resolution Short-Time Fourier Transform (STFT) loss is applied to a sub-batch of decoded audio waveforms to provide direct waveform-domain feedback, stabilizing spectral details without overpowering the pretrained audio prior.

3. Key Contributions

OmniVR Model: The first unified generative framework for joint audio-video restoration, adapting a 22B multimodal DiT from T2AV synthesis to AV2AV restoration.
OmniVRBench: The first benchmark specifically designed for joint audio-video restoration evaluation. It comprises 200 real historical film clips divided into:
- Real Historical Track (129 clips): Authentic archival footage with no ground truth, evaluating no-reference visual/audio quality, temporal consistency, and lip-sync.
- Controlled Degradation Track (71 clips): Talking-face clips with clean references, enabling full-reference and lip-sync evaluation.
Technical Innovations: The introduction of a joint degradation pipeline, an architecture-preserving T2AV-to-AV2AV transition with prompt annealing, and a first-frame anchoring strategy for long-form coherence.

4. Experimental Results

OmniVR was evaluated against state-of-the-art visual restoration methods (DeepRemaster, RealBasicVSR, MambaOFR), colorization methods (DDColor, ColorMNet), and audio restoration tools (VoiceFixer), as well as cascaded baselines.

Visual Quality: On the RTN old-film benchmark (no ground truth), OmniVR achieved the best scores on all six no-reference metrics (e.g., MUSIQ 64.77 vs. 52.18 for RealBasicVSR). On OmniVRBench, it surpassed all baselines on visual metrics (MUSIQ 61.87 vs. 52.38 for RealBasicVSR on the Real track).
Audio Quality: OmniVR achieved the best audio quality, with DNSMOS improving from 2.04 (degraded input) to 2.43 on the Real track, and FAD significantly reduced compared to VoiceFixer.
Synchronization: OmniVR demonstrated superior lip-sync performance (LSE-C 1.12 vs. 0.87 for VoiceFixer on the Real track), outperforming cascaded approaches which often degrade temporal alignment.
Colorization: Unlike baselines that either preserve grayscale or introduce tonal bias/flicker, OmniVR produced natural, temporally consistent colorization.
Human Evaluation: In a pairwise study with 12 annotators on 129 real clips, OmniVR was preferred 80.0% overall, with win rates of 82.5% for visual quality, 79.2% for audio, and 75.8% for sync, significantly outperforming the strongest baselines by over 50 percentage points.

5. Significance and Claims

The paper positions OmniVR as a milestone in audio-video enhancement, offering a new perspective that treats restoration as a coordinated generative task rather than independent enhancements. The authors claim that:

Joint Generation is Essential: Empirical analysis shows that historical film defects co-occur across modalities and that independent restoration can actively damage audio-visual consistency. Joint generation enables mutual promotion between modalities.
Generative Priors Enable Detail Recovery: By leveraging a 22B multimodal DiT, the model can reconstruct fine-grained details and natural colorization that regression-based methods miss, while the proposed conditioning mechanisms (anchoring, prompt annealing) ground the generation in the degraded source to avoid hallucination.
Practical Restoration: The model successfully restores long-form historical footage with coherent motion, natural color, and cleaned audio, addressing a gap in the literature where existing methods fail to handle the full audio-visual spectrum simultaneously.

The authors acknowledge limitations, such as the lack of clean references for real archival footage (necessitating proxy-based evaluation) and potential temporal drift in extremely damaged footage. They propose future work involving larger-scale paired real data and streaming (causal) generation for unbounded-length restoration. All code and model weights are planned for public release.