Technical Summary: Video Understanding Reward Modeling

Problem Statement

Despite significant advancements in multimodal reward modeling (MRMs) for text and image domains, progress in video understanding reward modeling remains severely constrained. The field faces two fundamental bottlenecks:

1. Lack of Robust Benchmarks: Existing benchmarks (e.g., MM-RLHF-RewardBench, VideoRewardBench) suffer from limited scale, insufficient coverage of video-specific dimensions (general, long, and reasoning tasks), and unreliable evaluation protocols. Crucially, they often lack Chain-of-Thought (CoT) reasoning traces, with average response lengths as short as 104 tokens, which misaligns with the "think-then-answer" paradigm required for complex video understanding. Furthermore, single-pass evaluation protocols are highly susceptible to position bias.
2. Scarcity of High-Quality Preference Data: While preference learning has advanced for video generation, high-quality, large-scale preference data for video understanding is scarce due to high annotation costs and construction complexity. Existing models trained on text or image-text data fail to transfer effectively to video reward tasks, often performing barely above random baselines.

Methodology

The authors propose a unified framework addressing benchmark design, data construction, and reward model training.

1. Video Understanding Reward Bench (VURB)

A new benchmark designed specifically for video understanding preference judgment, featuring:

• Scale and Scope: 2,100 preference pairs spanning three dimensions: General Video Understanding, Long Video Understanding, and Video Reasoning.
• Long CoT Reasoning: All preference responses include long reasoning traces, averaging 1,143 tokens, ensuring rich logical context.
• Robust Evaluation: Employs a Majority Voting protocol (N=8) where the order of chosen/rejected responses is randomly swapped to mitigate position bias.
• Data Quality: Responses are generated by five diverse multimodal models (including Qwen3VL and Gemini variants) and filtered via a two-stage process:
  • Length Consistency: Filters pairs with relative length differences >25% to prevent reward hacking based on verbosity.
  • Causal Consistency: Human experts verify that reasoning logically supports the final answer, removing cases where the answer is correct but reasoning is flawed (or vice versa).

2. Video Understanding Preference Dataset (VUP-35K)

To address data scarcity, the authors construct a large-scale dataset of 35,000 high-quality preference pairs via a fully automated, human-annotator-free pipeline:

• Generation: Prompts from existing benchmarks (e.g., Video-R1-CoT, Video-Holmes) are rolled out using candidate models.
• Verification: An LLM (Qwen3-4B) verifies the correctness of the final answer against ground truth.
• Augmentation Strategy: To overcome the issue of models answering most samples correctly (leading to a lack of negative samples), the authors introduce a random visual perturbation strategy. During rollout, video inputs are degraded (e.g., frame reduction, resolution reduction, dropout) to force models to produce incorrect responses, thereby enriching the diversity of reasoning failures.
• Final Corpus: VUP-35K is combined with 213K general image-text preference pairs to form a 245K training set.

3. Reward Models: VideoDRM and VideoGRM

Two distinct reward models are trained on the constructed data:

• VideoDRM (Discriminative): A standard reward model where the language model head is replaced with a scalar score head. It is trained using a Bradley-Terry ranking loss to maximize the margin between chosen and rejected responses.
• VideoGRM (Generative): A generative model trained via Group Relative Policy Optimization (GRPO). It produces both a preference decision and an interpretable reasoning trace. It is optimized using a binary reward signal (1.0 for correct identification of the chosen response, 0.0 otherwise).

Key Results

Performance on VURB

• Baseline Failure: Existing open-source models (including specialist models like R1-Reward and LLaVA-Critic-R1) perform poorly on VURB, with most falling below 55% accuracy (barely above the 50% random baseline). Proprietary models like GPT-5.2 achieve ~62.9%.
• Proposed Models:
  • VideoDRM achieves a state-of-the-art (SOTA) overall accuracy of 63.8%, outperforming GPT-5.2 and the largest Qwen3VL variants.
  • VideoGRM achieves 59.3%, emerging as the premier open-source generative reward model, surpassing Qwen3VL-32B-Thinking by 5.0%.

Generalization to VideoRewardBench

Both models achieve SOTA performance on the independent VideoRewardBench:

• VideoDRM: 64.7% overall accuracy.
• VideoGRM: 64.6% overall accuracy.
• Both models outperform high-tier proprietary baselines (e.g., Gemini-2.5-Pro at 63.6%) and significantly larger open-source models (e.g., LLaVA-Critic-72B at 63.0%).

Test-Time Scaling (Best-of-N)

Under Best-of-N evaluation on Video-Holmes and VideoMME:

• Both VideoDRM and VideoGRM provide significant gains over Self-Judge and Majority-of-N baselines.
• At N=8, VideoDRM reaches 49.70% accuracy (vs. 40.17% baseline), demonstrating their utility as reliable test-time reward signals for reranking.

Ablation Studies

• Impact of VUP-35K: Incorporating VUP-35K alone improves the backbone's accuracy from 52.4% to 62.5% on VURB, confirming that video-specific preference data is critical.
• Reasoning Capability: Generative reward training on VUP-35K alone improves downstream video understanding performance on benchmarks like Video-Holmes (+6.6% accuracy), indicating that the dataset enhances the model's inherent reasoning capabilities, not just its reward judgment.

Significance and Claims

The paper claims that the absence of specialized video understanding preference data and robust benchmarks has been the primary bottleneck for video reward modeling. By introducing VURB and VUP-35K, the authors demonstrate that:

1. Domain Specificity is Crucial: Models trained on text or image-text data cannot effectively transfer to video reward tasks; dedicated video preference data is essential.
2. CoT and Robust Evaluation Matter: Long reasoning traces and majority voting protocols are necessary to expose the true limitations of current models and ensure reliable evaluation.
3. Dual Paradigm Success: Both discriminative and generative training paradigms are effective for video reward modeling, with the proposed models setting new performance standards.
4. Foundation for Trustworthy Systems: The work provides a solid foundation for future research in trustworthy multimodal systems, highlighting the critical role of specialized data and training strategies in advancing video understanding.