VQ-Jarvis: Retrieval-Augmented Video Restoration Agent with Sharp Vision and Fast Thought

The paper introduces VQ-Jarvis, a retrieval-augmented intelligent agent for video restoration that leverages a new large-scale comparison dataset (VSR-Compare) to enhance degradation perception and employs a hierarchical scheduling strategy to efficiently determine optimal restoration trajectories for real-world heterogeneous degradations.

The Gist

Imagine you have an old, damaged home video. Maybe it's grainy, dark, blurry, and has rain streaks on the lens. You want to fix it, but the problem is that no single tool works for everything. A tool that fixes the darkness might make the rain look worse, and a tool that removes the blur might make the colors look weird.

Traditionally, video restoration software was like a one-size-fits-all suit. It tried to fix everything at once with a single, rigid formula. If the video was complex, the suit didn't fit, and the result looked bad.

This paper introduces VQ-Jarvis, a new kind of "Video Repair Agent." Think of VQ-Jarvis not as a single tool, but as a highly skilled, super-fast video repair team leader with two superpowers: "Sharp Vision" and "Fast Thought."

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blind" Repair Shop

Before VQ-Jarvis, existing AI repair tools were often "blind" to subtle differences.

• The Old Way: Imagine a repair shop that uses a generic ruler to measure quality. It might say, "This video looks 80% good," but it can't tell the difference between a video that is slightly better than another. It often picks the wrong fix because it doesn't truly "see" the subtle flaws.
• The Search Problem: Trying to find the perfect fix was like searching for a needle in a haystack by checking every single straw one by one. It took forever and wasted a lot of energy.

2. The Solution: VQ-Jarvis

VQ-Jarvis changes the game by acting like a smart detective who knows exactly what to do.

Superpower A: "Sharp Vision" (The Expert Eye)

To give the agent "sharp vision," the researchers built a massive new training library called VSR-Compare.

• The Analogy: Imagine they hired 20,000 pairs of human eyes and super-computers to watch thousands of "Before vs. After" video clips. They asked: "Which one looks better? Is it the brighter one, or the sharper one?"
• The Result: VQ-Jarvis learned from this massive library. Now, it can spot tiny differences that other AIs miss. It can tell the difference between a video that is "just okay" and one that is "perfectly crisp," just like a professional film editor can.

Superpower B: "Fast Thought" (The Smart Strategist)

Once VQ-Jarvis sees the problem, it needs to decide how to fix it. This is where its "Fast Thought" comes in. It uses a two-track strategy depending on how broken the video is:

• Track 1: The "Cheat Sheet" (For Easy Jobs)

  • Scenario: The video is just a little dark or slightly blurry.
  • Action: VQ-Jarvis looks at its "Cheat Sheet" (a database of past successful repairs). It says, "Hey, I've seen this exact type of rain-and-darkness before. I know the perfect fix! Let's just apply that."
  • Benefit: It's instant. No thinking required.

• Track 2: The "Chess Master" (For Hard Jobs)

  • Scenario: The video is a mess—dark, rainy, blurry, and low resolution all at once.
  • Action: The "Cheat Sheet" isn't enough. VQ-Jarvis switches to a step-by-step strategy. It tries a fix, checks if it got better, then tries the next tool. It's like a chess player thinking three moves ahead: "If I fix the rain first, then the blur, the colors will pop. But if I fix the blur first, the rain will look worse."
  • Benefit: It finds the perfect combination of tools without wasting time on dead ends.

3. The "Toolbox"

VQ-Jarvis doesn't invent new tools; it's the manager of a toolbox filled with the best video repair tools available (like specialized AI for removing rain, fixing low light, or sharpening images).

• The Magic: It knows which tool to use and in what order.
  • Example: It knows you must remove the rain before you brighten the image, otherwise, the rain streaks will look like weird shadows. It figures this out automatically.

Why This Matters

In the real world, videos are rarely just "blurry." They are a messy mix of problems.

• Old AI: Tries to fix everything with one big hammer. Often breaks things.
• VQ-Jarvis: Looks at the mess, identifies the specific problems, grabs the right tools from its toolbox, and fixes them in the perfect order, all in the blink of an eye.

In short: VQ-Jarvis is the difference between a robot that blindly follows a recipe and a master chef who tastes the food, adjusts the spices, and knows exactly when to add the salt to make the dish perfect. It makes video restoration faster, smarter, and much higher quality.

Technical Summary

1. Problem Statement

Real-world video restoration faces significant challenges due to heterogeneous and complex degradations (e.g., mixed noise, blur, compression, low-light, and rain) that vary temporally across frames. Existing approaches suffer from three main limitations:

• Static Architectures: Traditional methods (task-specific or "all-in-one" networks) rely on fixed inference pipelines, lacking the adaptability to handle diverse, coupled degradation patterns in real-world scenarios.
• Insufficient Quality Perception: Existing agent-based systems often rely on generic quality metrics (e.g., CLIPIQA, ManIQA) trained on natural images. These metrics fail to detect subtle differences between paired restoration results, leading to sub-optimal decision-making.
• Inefficient Search Strategies: Current agent frameworks often use greedy, step-by-step trial-and-error search, which is computationally expensive and slow, failing to leverage the fact that videos with similar degradations often share optimal restoration strategies.

2. Methodology: VQ-Jarvis Framework

The authors propose VQ-Jarvis, a retrieval-augmented, all-in-one intelligent agent designed with "Sharp Vision" (accurate perception) and "Fast Thought" (efficient search).

A. VSR-Compare Benchmark (The Foundation)

To address the lack of data for training agents to judge subtle quality differences, the authors constructed VSR-Compare, the first large-scale video paired enhancement dataset.

• Scale: ~20,000 comparison pairs covering 7 degradation types, 11 enhancement operators, and diverse domains (AIGC, human faces, natural scenes).
• Annotation Strategy: A Human-Machine Collaborative approach. Multiple MLLM experts (Qwen-MoE, Gemini3 Pro, GPT-4o) generate initial preferences and reasoning. Inconsistent items are filtered, and reasoning is fused. Human experts then verify and annotate a subset, resulting in high-quality labels.
• Outcome: This dataset enables the training of a Multi-operator Judge Model that achieves 93% agreement with human experts, far surpassing existing MLLMs.

B. Core Components

1. Degradation Perception Model ($H_{deg}$):
   • Estimates a scalar quality score and a degradation attribute vector (e.g., presence of rain, low-light).
   • Trained using Group Relative Policy Optimization (GRPO) with hybrid rewards (continuous error, binary accuracy, and degradation recognition).
2. Multi-operator Judge Model ($H_{judge}$):
   • A preference comparison model trained to select the best restoration result among candidates based on visual quality, temporal consistency, and dynamic degree.
   • Also trained via GRPO using the VSR-Compare dataset.
3. Hierarchical Operator Scheduling Strategy:
   The agent adaptively selects a restoration strategy based on the predicted difficulty (quality score) of the input video:
   • One-Step Retrieval (Easy Cases): For videos with lower degradation difficulty, the agent queries a Retrieval-Augmented Generation (RAG) library. This library contains pre-computed optimal restoration trajectories for 1,000 reference videos. The system uses quality-aligned embeddings (aligning CLIP features with quality reasoning) to find the most similar video and retrieves its optimal trajectory.
   • Step-wise Greedy Search (Hard Cases): For complex degradations, the agent performs a sequential search. It decomposes the task into sub-tasks (e.g., deraining, low-light enhancement, super-resolution). At each step, it applies candidate operators, evaluates them using the Judge Model, and selects the best one (tournament-style elimination).

3. Key Contributions

1. VQ-Jarvis Agent: A novel framework integrating quality-aware perception, preference-based comparison, and adaptive scheduling, outperforming static models and existing agents.
2. VSR-Compare Dataset: The first large-scale video paired enhancement dataset (20K pairs) constructed via human-MLLM collaboration, enabling robust training of preference and perception models.
3. Hierarchical Scheduling: A hybrid strategy that balances efficiency and accuracy by combining RAG-based retrieval for simple cases and greedy search for complex ones.
4. Reconstruction Order Experience: The paper empirically derives optimal operator ordering (e.g., Deraining $\to$ Low-light $\to$ BNC/SR $\to$ Frame Interpolation) to maximize quality and temporal consistency.

4. Experimental Results

The authors evaluated VQ-Jarvis on real-world super-resolution (UDM10, YouHQ40) and multi-degradation restoration benchmarks.

• Performance: VQ-Jarvis consistently outperforms state-of-the-art methods (SeedVR2, DOVE, FlashVSR, MGLD-VSR) and metric-guided baselines (DOVER-guide).
  • Metrics: It achieves superior scores in perceptual metrics (CLIPIQA, DOVER, VQ-Insight) and distortion metrics (LPIPS, DISTS).
  • Degradation Detection: The perception model achieves 91.53% average accuracy in detecting degradation types, significantly beating general MLLMs (e.g., Qwen3-MoE at 66.99%).
• Efficiency:
  • The hybrid strategy significantly reduces inference time compared to pure greedy search.
  • Time Complexity: The RAG-based path is near-linear $O(T_{det} + T_{rag} + K \cdot T_{op})$, while the greedy path is $O(K \cdot (N \cdot T_{cmp} + T_{max}))$. The adaptive switching ensures a favorable trade-off, achieving high quality with moderate inference time (~17.88s vs. ~42.65s for pure greedy).
• Ablation Studies:
  • Removing the RAG component (using only greedy search) improves quality but drastically increases time.
  • Using standard CLIP embeddings for retrieval instead of quality-aligned embeddings results in lower restoration quality.

5. Significance

This work represents a paradigm shift in video restoration from static, end-to-end models to dynamic, decision-centric agents.

• Generalization: By treating restoration as a combinatorial optimization problem guided by an agent, the system generalizes better to unseen, complex real-world degradations.
• Data-Centric AI: The construction of VSR-Compare highlights the critical need for high-quality, preference-aligned datasets to train agents that can perceive subtle visual nuances.
• Efficiency: The retrieval-augmented approach demonstrates that leveraging prior knowledge (retrieving optimal trajectories) can drastically reduce the computational cost of agent-based reasoning without sacrificing quality.

In conclusion, VQ-Jarvis establishes a new standard for intelligent video restoration, proving that agents equipped with "sharp vision" (specialized perception models) and "fast thought" (retrieval-augmented reasoning) can effectively solve the complex, heterogeneous challenges of real-world video degradation.