Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLMReward Models

1. Problem Statement

Vision-Language Models (VLMs) increasingly rely on Generative Reward Models (GRMs) to evaluate outputs. Unlike scalar reward models, GRMs produce structured critiques consisting of three parts:

1. Rubric ($R$): A set of evaluation criteria.
2. Evaluation ($E$): A criterion-by-criterion assessment.
3. Verdict ($A$): The final preference decision.

The Core Issue: Existing GRM training pipelines optimize exclusively for the final verdict ($A$). The intermediate rubric ($R$) is treated as an incidental byproduct, leaving it unsupervised and unverified. This leads to:

• Post-hoc Rationalization: Models may generate rubrics that merely justify a pre-determined answer rather than guiding a principled evaluation.
• Poor Transferability: Rubrics generated by the model often fail when applied by independent evaluators or on different task distributions.
• Limitations of Current Fixes: The common "LLM-as-a-Judge" approach is computationally expensive and, crucially, provides non-differentiable signals, preventing the rubric quality from being optimized via Reinforcement Learning (RL).

2. Methodology: Proxy-GRM

The authors propose Proxy-GRM, a framework that introduces a closed-loop training mechanism to explicitly optimize rubric quality using a dedicated Proxy Agent.

A. Core Concept: Rubric Transferability

The paper defines Rubric Transferability as the ability of a generated rubric to guide an independent agent to the correct preference verdict. If an external agent can correctly judge a pair of responses using only the generated rubric (and the original inputs), the rubric is considered high-quality and transferable.

B. The Proxy Agent

A lightweight Proxy Agent is trained specifically to consume rubrics, not generate them.

• Input: Original query ($q$), image ($I$), candidate responses ($r_1, r_2$), and a generated rubric ($R$).
• Output: A verdict ($A \in \{1, 2\}$).
• Training: The proxy is trained via Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to faithfully follow provided rubrics.
• Role: It acts as a verifier. Its prediction accuracy serves as a rubric-quality reward signal for the main policy model.

C. Training Pipeline

The framework involves two main stages:

1. Proxy Agent Training:

   • Proxy-SFT: Fine-tuned on 5k high-quality samples to learn to follow rubrics.
   • Proxy-RL: Further trained on 10k samples to improve robustness.
   • Key Finding: Proxy-SFT (SFT-based) outperforms Proxy-RL (RL-based) as a verifier, suggesting that outcome-only RL optimization can degrade the process-level fidelity required for evaluation.

2. Policy Model (GRM) Training:

   • Cold-Start: The base VLM is SFT-trained on 10k samples to learn the output format.
   • Reinforcement Learning (GRPO): The policy is optimized using a composite reward function:
     $$r = r_{acc} + r_{proxy} + 0.5 \cdot r_{format}$$
     • $r_{acc}$: Accuracy of the final verdict.
     • $r_{proxy}$: The novel component. +1 if the frozen proxy agent agrees with the ground truth using the generated rubric; -1 otherwise.
     • $r_{format}$: Adherence to XML structure.

3. Key Contributions

1. Identification of Rubric Transferability: The paper identifies rubric transferability as a critical, under-explored dimension of GRM quality and proposes a principled solution.
2. Proxy-Guided Closed-Loop Training: Introduces a framework where an independent proxy agent provides a differentiable reward signal for rubric quality, solving the "open-loop" limitation of LLM-as-a-Judge.
3. Surprising Insight on Proxy Agents: Demonstrates that SFT-based proxy agents are superior to RL-based ones for evaluation tasks, revealing a tension between outcome-level optimization and process-level evaluation fidelity.
4. Data Efficiency: Achieves State-of-the-Art (SOTA) results with only ~50k training samples, which is 4x less data than comparable methods (e.g., R1-Reward, Unified-Reward) trained on >200k samples.

4. Experimental Results

The method was evaluated on three major benchmarks: VL-RewardBench, Multimodal Reward Bench, and MM-RLHF-Reward Bench.

• Performance: Proxy-GRM (using Proxy-SFT) achieved SOTA results across all benchmarks.
  • VL-RewardBench: 75.22% accuracy (vs. 73.8% for the previous best, Unified-Reward-Think).
  • Multimodal Reward Bench: 85.62% accuracy (outperforming closed-source models like Claude-3.7-Sonnet and GPT-4o).
  • MM-RLHF-Reward Bench: 82.94% accuracy.
• Data Efficiency: The model outperformed baselines trained on 4x more data, proving that proxy-guided verification is a more efficient learning signal than simple data scaling.
• Transferability: Rubrics generated by Proxy-GRM significantly improved the performance of unseen evaluator models (e.g., when Unified-Reward-SFT used Proxy-GRM's rubrics, its accuracy jumped by ~3.76 points). This confirms the rubrics capture universal evaluation principles rather than model-specific shortcuts.
• Ablation Studies:
  • Proxy Selection: Proxy-SFT > Proxy-RL. Larger general models (32B) performed worse than the specialized 7B Proxy-SFT, highlighting the value of targeted training.
  • Multi-Agent Ensembles: Combining multiple proxy agents degraded performance due to conflicting reward signals, suggesting a single, well-calibrated proxy is superior to an ensemble of heterogeneous agents.
  • Reward Aggregation: Simple implicit addition of rewards ($r_{acc} + r_{proxy}$) outperformed complex schemes that explicitly modified accuracy rewards based on proxy feedback.

5. Significance

This work fundamentally shifts the paradigm of training reward models from outcome-only supervision to process supervision. By treating the rubric as a first-class citizen that must be independently verifiable, the authors:

• Eliminate "rationalization" where models fake reasoning to match a desired answer.
• Create evaluation criteria that are robust, interpretable, and transferable across different models and tasks.
• Demonstrate that high-quality reasoning artifacts can be learned efficiently without massive datasets, provided the training loop includes a mechanism to verify the quality of the reasoning process itself.

The code is available at: https://github.com/Qwen-Applications/Proxy-GRM.