💻 computer science

VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning

The paper introduces VR-Thinker, a novel video reward model framework that enhances reasoning fidelity and reliability by enabling active visual evidence acquisition through a "thinking-with-image" mechanism and a multi-stage reinforcement fine-tuning pipeline, achieving state-of-the-art performance on video preference benchmarks.

Original authors: Qunzhong Wang, Jie Liu, Jiajun Liang, Yilei Jiang, Yuanxing Zhang, Yaozhi Zheng, Xintao Wang, Pengfei Wan, Xiangyu Yue, Jiaheng Liu

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qunzhong Wang, Jie Liu, Jiajun Liang, Yilei Jiang, Yuanxing Zhang, Yaozhi Zheng, Xintao Wang, Pengfei Wan, Xiangyu Yue, Jiaheng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a film critic trying to judge two short movies. Your job is to decide which one is better based on a specific description (like "a dog chasing a ball").

The Problem with Current Critics (Old Reward Models):
Most current AI critics are like critics who are forced to watch a movie through a tiny keyhole.

The Keyhole Limit: Because their "brain" (computer memory) is small, they can only peek at a few frames of the movie at the start. If the movie is long, they miss the best parts or the subtle details.
The "Static" Memory: Once they take that first peek, they have to close the keyhole. They have to write their review based only on those first few glimpses and their memory. If they forget a detail or imagine something that wasn't there (a "hallucination"), they can't go back to check. They just have to guess.

The Solution: VR-Thinker (The New Super-Critic):
The paper introduces VR-Thinker, a new kind of AI critic that doesn't just peek; it thinks with images.

Instead of being stuck with a static memory, VR-Thinker is like a critic with a remote control and a magnifying glass.

How VR-Thinker Works (The Analogy)

1. The "Active Detective" Approach
When VR-Thinker starts watching, it sees a few frames. But instead of immediately writing a review, it asks itself: "Wait, I'm not sure about the dog's tail movement. I need to look closer."

The Tool: It uses a special tool called "Select Frame." It can say, "Show me frames 12, 45, and 89."
The Result: It actively goes back into the video, grabs the specific moments it needs, and updates its understanding. It doesn't just guess; it gathers evidence.

2. The "Sliding Window" Memory
Imagine your desk is getting cluttered with photos. If you keep adding photos, you run out of space.

The Trick: VR-Thinker uses a "Sliding Window." It keeps the most recent photos it's looking at on the desk. As it looks at new photos, it summarizes the old ones into a short note (like a sticky note) and slides the old photos off the desk.
Why it matters: This lets it watch a very long movie without running out of memory. It can keep investigating new details without forgetting the whole story.

3. The "Thinking Process" (Chain of Thought)
Old critics just shout out a score. VR-Thinker talks to itself first.

It says: "I see the dog running, but the background looks blurry. Let me check frame 50 to see if the blur is real or just a glitch."
It checks the frame.
It updates its thought: "Okay, the blur is real. The video quality is lower."
It only gives the final score after this careful investigation.

How They Taught the AI to Think (The Training)

The researchers didn't just tell the AI to do this; they trained it in three clever stages, like training a puppy:

Cold Start (The Lesson): They showed the AI examples of how to be a good detective. They taught it the rules: "If you are unsure, use your remote control to pick more frames. Write down your thoughts in this specific format."
Rejection Sampling (The Filter): They let the AI practice on thousands of videos. If the AI got the answer right and used the right reasoning steps, they kept that lesson. If it got it right by luck or used bad reasoning, they threw it away. They only kept the "perfect" examples to teach the AI further.
The "Group Game" (GRPO): Finally, they played a game where the AI generated several different ways to judge the same video. They compared them: "Which one looked at the most details? Which one was most logical?" They rewarded the AI for the best reasoning path, encouraging it to always dig deeper.

Why This Matters

Better for Long Videos: Old critics fail at long movies because they forget the beginning. VR-Thinker can watch the whole thing, zooming in on the important parts as needed.
Fewer Mistakes: Because it can go back and check its work, it makes fewer "hallucinations" (imagining things that aren't there).
The Future: This proves that giving AI the ability to "look again" and "think with pictures" makes it much smarter at understanding video, which is crucial for building better AI video generators in the future.

In short: VR-Thinker is the difference between a critic who glances at a movie poster and guesses the plot, versus a critic who watches the whole film, pauses to check the details, and writes a review based on actual evidence.

: Contains the reasoning chain. * : Intermediate judgments with confidence levels. * : Final judgment. * Tool calls (`): To request specific frames.

B. Three-Stage Training Pipeline

VR-Thinker is trained using a progressive pipeline to instill and refine visual reasoning:

Cold Start (Supervised Fine-Tuning):
- Uses curated high-quality CoT data generated by a powerful model (GPT-4o) to teach the model the specific reasoning format and tool invocation syntax.
- Ensures the model learns to generate valid tool calls and structured reasoning before reinforcement learning.
Rejection Sampling Fine-Tuning (RFT):
- The model generates multiple CoT traces on large-scale preference datasets.
- Strict Filtering: Only traces where every per-dimension judgment and the overall judgment match the ground truth are retained.
- The model is fine-tuned on these high-quality "verified" traces to consolidate reasoning skills and increase the yield of correct reasoning paths.
Group Relative Policy Optimization (GRPO):
- A reinforcement learning stage that optimizes the model toward high-quality reasoning.
- Reward Design:
  - Accuracy Reward: Combines Overall Preference and Per-Dimension Judgments (e.g., Text Alignment, Visual Quality, Motion Quality). This expands the answer space, reducing the risk of rewarding "lucky" guesses that get the final answer right but for the wrong reasons.
  - CoT Gain Reward: Incentivizes the model to improve its accuracy as it gathers more visual evidence (rewarding the delta between intermediate and final answers).
  - Exploratory Incentive: A constraint-based reward (via Lagrangian Relaxation) that forces the model to perform a minimum proportion of multimodal reasoning (tool calls) to prevent it from reverting to lazy, text-only reasoning.

3. Key Contributions

First Visual-Reasoning RM: Introduces VR-Thinker, the first reward model that can actively retrieve and update visual evidence during the reasoning process, breaking the "static prompt" limitation.
Thinking-with-Image Framework: Proposes a novel paradigm where the RM acts as an agent that can "zoom in" on videos, effectively removing the upper bound on the number of frames it can process.
Advanced Training Strategy: Develops a robust three-stage pipeline (Cold Start $\to$ Rejection Sampling $\to$ GRPO) with specialized reward signals (CoT Gain, Exploratory Incentive) to ensure the model learns how to reason visually, not just what to answer.
Efficient Long-Video Handling: Demonstrates that by using a sliding memory window and linguistic summarization, the model can evaluate long videos with high fidelity without the prohibitive memory costs of feeding all frames into the context.

4. Experimental Results

The model was evaluated on three major video preference benchmarks: GenAI-Bench, VideoGen-Reward, and MJ-Bench-Video.

State-of-the-Art Performance: The 7B parameter VR-Thinker achieved SOTA results among open-source models:
- 80.5% on VideoGen-Reward (diff).
- 82.3% on GenAI-Bench (diff).
- 75.6% on MJ-Bench-Video (diff).
- This represents an average improvement of 11.4% over previous baselines.
Long Video Superiority: On "hard" subsets (long videos and complex prompts), VR-Thinker showed significantly smaller performance drops compared to baselines. While standard models degrade as video length increases due to information loss, VR-Thinker maintained high accuracy by retrieving missing frames.
Ablation Studies:
- Removing visual reasoning (random frame retrieval) caused a sharp performance drop, proving the necessity of reasoning-driven retrieval.
- The Rejection Sampling stage was crucial for providing a high-quality foundation for GRPO.
- CoT Gain Reward and Exploratory Incentive were essential for preventing the model from skipping visual steps.
Efficiency: Despite longer inference chains, VR-Thinker maintains stable GPU memory usage (due to the window mechanism) and achieves higher throughput than naive long-context models that try to load all frames at once.

5. Significance

VR-Thinker addresses a fundamental bottleneck in video generation alignment: the inability of current reward models to handle long, complex videos without losing detail. By shifting from a passive "read-once" paradigm to an active "investigate-and-retrieve" paradigm, it significantly improves the fidelity and reliability of reward signals. This enables more precise alignment of video generation models with human preferences, particularly for long-form content where temporal consistency and fine-grained details are critical. The work establishes a new direction for multimodal reasoning where vision is treated as a dynamic, queryable resource rather than a static input.