💻 computer science

V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning

This paper introduces V-Rubrics, a reinforcement learning framework that enhances the visual faithfulness of vision-language models by decomposing responses into atomic propositions for structured, partial-credit scoring across visual grounding, reasoning, and instruction following, thereby overcoming the limitations of scalar outcome rewards.

Original authors: Shulin Tian, Minglun Li, Yuhao Dong, Hao Ding, Jiarui Yao, Haiwen Diao, Jingkang Yang, Hongyuan Zhu, Ziwei Liu

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shulin Tian, Minglun Li, Yuhao Dong, Hao Ding, Jiarui Yao, Haiwen Diao, Jingkang Yang, Hongyuan Zhu, Ziwei Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a specific type of computer program has emerged that can look at a photograph and describe what it sees. These systems, known as vision-language models, are trained to connect the pixels of an image with the words of human language. They can identify a dog in a park, read a menu in a foreign language, or explain a complex chart. However, a persistent problem has plagued these systems: they are often fluent but unfaithful. A model might generate a perfectly smooth sentence that sounds confident, yet it could be describing an object that isn't there, misreading a number on a graph, or drawing a conclusion that the picture simply does not support. For a human, this is a hallucination; for the machine, it is a failure to anchor its words in visual reality. The core challenge for researchers has been how to teach these machines to be honest about what they see, rather than just good at guessing the right answer.

The researchers behind this study approached this problem by realizing that the way they currently reward these machines is too blunt. In standard training, a computer is shown an image and a question, and it produces an answer. If the answer is correct, the system gets a point; if it is wrong, it gets nothing. This method works well for simple tasks, but it fails when the reasoning process is complex. Imagine a student who correctly identifies the objects in a photo but then makes a logical error in connecting them, or a student who gets the final answer right by sheer luck despite misreading the chart. A simple "right or wrong" score cannot tell the difference between these scenarios. It cannot see that the student saw the right things but thought about them incorrectly, or that they missed a specific instruction while getting the main point. Without this nuance, the machine learns to prioritize the final outcome over the truthfulness of the steps taken to get there.

To solve this, the team introduced a new training method that treats the answer not as a single block of text, but as a collection of smaller, checkable facts. They broke down the ideal response into a list of specific requirements, or "rubrics." For a question about a diagram of the solar system, the rubric would not just ask for the final answer. Instead, it would list distinct steps: "Did the model correctly identify the sun?" "Did it notice that the moon is in a straight line with the earth?" "Did it explain why this alignment causes high tides?" Each of these steps is assigned a specific weight based on its importance. The system then checks the computer's response against each of these tiny criteria individually. If the model gets the object identification right but fails the reasoning step, it receives partial credit for the first part and a penalty for the second. This allows the training process to see exactly where the machine went wrong and to reward it for the parts it got right, even if the final conclusion was flawed.

The researchers built a massive dataset to teach their system this new way of thinking. They gathered over 50,000 examples from 17 canonical sources covering diagram reasoning, chart understanding, document VQA, mathematical visual reasoning, counting, educational QA, and general visual reasoning. For each example, they used a powerful language model to generate these detailed rubrics, turning a simple question-and-answer pair into a structured lesson plan. They then trained their vision model using a technique called reinforcement learning, where the computer plays the role of a student trying to improve. Instead of waiting for a final grade, the student gets immediate feedback on every sentence it writes. If it describes a visual element accurately, it earns points. If it hallucinates a detail or skips a logical step, it loses points. Crucially, the system learns to associate these points with the specific part of the text where the action happened, though this localization is approximate and relies on fuzzy matching to align the feedback with the response, so it knows which words to keep and which to change.

The results of this approach were significant, particularly for tasks that require careful observation and logical deduction. When tested on a wide range of benchmarks, the model trained with these detailed rubrics outperformed both the standard version and a version trained only on final answers. The improvement was most noticeable in areas like visual mathematics and chart analysis, where a single unsupported claim can ruin the entire solution. In these tests, the rubric-trained model was better at preserving the chain of reasoning, ensuring that every conclusion was backed by evidence visible in the image. It learned to avoid the trap of guessing the right answer for the wrong reasons. The study suggests that by breaking down the concept of "correctness" into smaller, verifiable pieces, we can guide artificial intelligence toward a more reliable and truthful understanding of the visual world, moving beyond simple fluency to genuine comprehension.

Technical Summary: V-Rubrics: Visual Faithfulness via Rubric-Based Reinforcement Learning

Problem Statement

Vision-language models (VLMs) frequently generate fluent responses that lack sufficient grounding in visual evidence. A single unsupported object, incorrect chart value, or invalid intermediate inference can undermine an otherwise plausible response. The authors identify this not merely as an evaluation failure, but as a credit-assignment problem in multimodal post-training.

Standard reinforcement learning (RL) alignment pipelines typically reduce a response to a scalar outcome reward (e.g., binary correctness or a holistic judge score). While effective for tasks with directly verifiable answers, these scalar rewards fail in visual reasoning scenarios where a response may contain a mix of correct observations, invalid inferences, and format compliance. A single outcome signal cannot distinguish which visual facts are grounded, which reasoning steps are valid, or which instruction constraints are missed. Consequently, the model cannot learn to correct specific local errors without potentially sacrificing global correctness.

Methodology

The paper proposes Visual Rubrics-Based Reinforcement Learning (V-Rubrics), a framework that decomposes reference responses into atomic propositions to provide fine-grained, structured supervision.

1. V-Rubrics 50K Dataset Construction

To support this framework, the authors constructed V-Rubrics 50K, a training set of 50,248 examples derived from 17 visually grounded sources (including diagram reasoning, chart understanding, document VQA, and mathematical visual reasoning).

Data Filtering: Sources underwent rule-based filtering for validity, non-triviality, and language quality.
Difficulty Stratification: A fine-tuned policy ( $\pi_{SFT}$ ) generated eight rollouts per example. The empirical success rate ( $k/8$ ) was used to assign difficulty labels (Hard, Medium, Simple), resulting in a stratified mixture of 36.1% hard, 50.4% medium, and 13.6% simple examples.
Rubric Annotation: Each example was annotated by Gemini-3-Pro using a structured prompt to decompose the reference response into atomic criteria across three dimensions:
- Visual Faithfulness (VF): Checks if stated content is supported by image evidence (objects, attributes, relations, counts, text).
- Reasoning Consistency (RC): Checks if conclusions follow logically from observed visual evidence.
- Instruction Following (IF): Checks if the response satisfies prompt constraints.
Weighting: Criteria are assigned importance weights (Essential, Important, Optional, Pitfall) and numeric weights to determine their contribution to the reward.

2. Training Framework

The training pipeline utilizes Group Relative Policy Optimization (GRPO) starting from a supervised fine-tuned (SFT) checkpoint of Qwen3-VL-8B-Instruct, adapted from the OpenMMReasoner cold-start recipe.

Reward Design:
- Rubric Reward ( $R_{rub}$ ): An LLM verifier independently scores each atomic criterion as satisfied (1) or not (0). Crucially, the verifier receives the generated response and the criterion text, not the raw image. Positive criteria provide importance-weighted partial credit. A "Pitfall" violation acts as a semantic veto.
- Hybrid Reward: The final reward combines the rubric signal with a standard answer-level reward ( $R_{ans}$ ) using a weighting factor $\alpha$ : $R = \alpha R_{ans} + (1-\alpha) R_{rub}$ .
Prefix-Localized Credit:
- Unlike standard GRPO which assigns a single advantage to the entire sequence, V-Rubrics localizes advantages.
- When a verifier identifies a supporting evidence span for a specific rubric item, the advantage for that item is broadcast only to the response tokens up to the end of that span ( $t_{j,end}$ ).
- The total advantage $A_t$ at token $t$ is a sum of the global answer advantage and the prefix-masked rubric advantages:
  $A_t = \alpha z(R_{ans}) + (1-\alpha) \sum_{j} \rho_j z(s_j) \mathbb{I}(t \le t_{j,end})$
- This allows the model to receive credit for grounded intermediate steps even if the final answer is incorrect, and penalize specific hallucinations without collapsing the entire sequence's gradient.

Key Contributions

Formulation of Visual Faithfulness as a Credit-Assignment Problem: The paper reframes visual grounding from a post-hoc diagnostic into a structured training interface. It introduces a method to preserve item-level rubric components and localize their advantages to response prefixes when evidence spans are available.
V-Rubrics 50K Resource: The creation of a 50,248-example dataset that converts reference responses into visually grounded atomic criteria with explicit capability dimensions (VF, RC, IF) and importance weights.
Empirical Validation of Rubric-Based GRPO: The study demonstrates that rubric-based GRPO outperforms both the shared SFT baseline and scalar answer-only GRPO. The improvements are most pronounced on benchmarks requiring grounded intermediate reasoning.

Experimental Results

The authors evaluated their approach on 10 benchmark families covering general VLM ability, knowledge-oriented reasoning, visual math, chart reasoning, and logic.

General & Knowledge Benchmarks: Rubric-based GRPO achieved an Overall Average of 68.04, outperforming the SFT baseline (64.93) and scalar answer-only GRPO (66.25). The gains were particularly significant on MMMU and MMMU-Pro, suggesting rubrics aid multi-step academic reasoning.
Visual Math & Logic: On visual-reasoning tasks (MathVision, DynaMath, WeMath, LogicVista), the rubric-based model improved the Overall Average by 4.00 points over the SFT baseline (62.45 vs. 58.45), surpassing the scalar GRPO variant (61.94).
Ablation Studies: Comparing reward designs showed that component-wise, prefix-localized credit (68.04) outperformed scalar sequence-level rubric aggregation (67.74), confirming that localizing credit to specific reasoning steps is more effective than aggregating rubric scores at the sequence level.
Qualitative Analysis: Case studies showed that rubric-trained models were better at preserving intermediate reasoning steps (e.g., explicitly tracing graph segments before answering) and avoiding unsupported inferences compared to answer-only trained models.

Significance and Claims

The paper claims that rubrics serve as a useful reward abstraction for visual post-training. By decomposing responses into atomic, visually grounded propositions, the framework addresses the "credit-assignment failure" inherent in scalar rewards.

The authors argue that future VLM post-training should treat visual grounding not just as an evaluation metric, but as a structured training interface that explicitly exposes where reasoning succeeds or fails. The results suggest that preserving item-level judgments and localizing credit to specific response prefixes allows models to learn from partial successes and specific failures, leading to more robust visual reasoning capabilities, particularly in complex, multi-step tasks.

The paper remains modest regarding its limitations, acknowledging that the dataset relies on automatically generated rubrics and judge-model verification, which may introduce bias or reference-answer bias. Additionally, prefix-credit localization is an approximation based on fuzzy matching of verifier sentences, rather than exact token-level supervision.