Self-Rewarding Vision-Language Model via Reasoning Decomposition

1. Problem Statement

Vision-Language Models (VLMs) currently face two critical limitations in their reasoning capabilities:

• Visual Hallucinations: Generating descriptions or answers inconsistent with the visual input.
• Language Shortcuts: Bypassing visual understanding entirely and relying on text priors (language shortcuts) to guess the correct answer.

Existing post-training methods, particularly Reinforcement Learning (RL) approaches like R1-style models, often exacerbate these issues. They typically rely on final answer matching as the sole reward signal. This sparse supervision fails to guide the intermediate visual reasoning process, causing models to prioritize language-based reasoning over visual perception. Furthermore, methods that attempt to add visual supervision often rely on:

• Human Annotations: Which are labor-intensive, costly, and difficult to scale.
• External Reward Models: Which introduce high latency, require additional GPU resources, and suffer from distributional shifts or biases inherited from the distillation source.

2. Methodology: Vision-SR1

The authors propose Vision-SR1, a three-stage self-rewarding RL framework that improves visual reasoning without external supervision. The core innovation is the decomposition of reasoning into distinct visual and language components, verified by the model itself.

A. Reasoning Decomposition (See-Think Format)

The model is prompted to generate a structured output separating three components:

1. Visual Reasoning ($c$): A self-contained visual description capturing all necessary details from the image to solve the task.
2. Language Reasoning ($t$): Chain-of-Thought (CoT) reasoning based on the visual description.
3. Answer ($a$): The final response.

B. The Three-Stage Training Process

The training involves two rollout passes and a specific optimization objective:

1. First Pass (Standard Rollout):

   • Input: Image ($i$) + Query ($q$).
   • Output: Structured response ($c, t, a$).
   • Reward: An Answer Reward ($r_{ans}$) is computed based on the correctness of the final answer against the ground truth.

2. Second Pass (Self-Reward Rollout):

   • Input: Query ($q$) + Generated Visual Reasoning ($c$) only. The original image is excluded.
   • Process: The model attempts to derive the correct answer using only the text description generated in the first pass.
   • Reward: A Visual Reward ($r_{visual}$) is assigned. If the model can correctly answer the question using only the generated description, the description is deemed "self-contained" and "faithful." This acts as an intrinsic reward for accurate visual perception.

3. Multi-Reward Policy Optimization:

   • Instead of simply summing rewards (which entangles signals), Vision-SR1 uses a Multi-Reward Policy Optimization strategy based on Group Relative Policy Optimization (GRPO).
   • Decoupled Advantages: The advantage ($\hat{A}$) and log-probabilities are calculated separately for the answer generation and the visual reasoning generation.
   • Loss Function: The total loss combines the policy gradients for both rewards ($L_{actor}$) and separate KL divergence penalties ($L_{KL}$) for each component. This ensures clear gradient attribution, allowing the model to optimize visual perception and language reasoning independently.

C. Computational Efficiency

• No External Models: Vision-SR1 does not require external reward models or human annotators.
• No Extra GPUs: The self-reward mechanism reuses the policy model itself, avoiding the overhead of hosting separate judge models.
• Overhead: The two-stage rollout adds only ~10-15% computational overhead compared to standard single-stage GRPO, making it highly scalable.

3. Key Contributions

1. Self-Rewarding Framework: Introduces a method where the VLM validates its own visual grounding by re-prompting itself with the generated description, eliminating the need for external visual supervision.
2. Reasoning Decomposition: Explicitly isolates visual perception from language reasoning, forcing the model to produce self-contained visual descriptions before reasoning.
3. Multi-Reward Policy Optimization: Proposes a decoupled loss function that computes advantages and KL penalties separately for visual and language components, preventing reward entanglement and enabling fine-grained optimization.
4. Language Shortcut Rate (LSR) Metric: Introduces a new metric to quantify how often a model answers correctly despite having incorrect or non-self-contained visual reasoning, providing a way to measure reliance on language shortcuts.

4. Experimental Results

The method was evaluated on diverse benchmarks using Qwen2.5-VL (3B and 7B) and Mimo-VL-7B backbones.

• Performance Gains: Vision-SR1 outperformed baselines (including Vision-R1 and Perception-R1) across general visual understanding (MMMU, RealWorldQA), mathematical reasoning (MathVerse, MATH-Vision), and hallucination detection (HallusionBench).
  • Example: On the 7B backbone, Vision-SR1 achieved an average score of 52.2, compared to 44.5 for Vision-R1.
• Hallucination Mitigation: The model significantly reduced visual hallucinations and improved robustness against language shortcuts.
• LSR Reduction: Vision-SR1 demonstrated a lower Language Shortcut Rate (LSR) compared to baselines, indicating a stronger reliance on genuine visual perception rather than text priors.
• Ablation Studies:
  • Removing the self-reward mechanism led to a drop in performance, confirming the necessity of the visual reward signal.
  • The method improved spatial reasoning (MMSI-Bench, OmniSpatial) and general knowledge tasks without degrading text-only capabilities as severely as other RL methods.
• Attention Analysis: Layer-wise analysis showed that post-training increased attention to visual tokens in early feature extraction and late integration layers, confirming the model learned to better process visual information.

5. Significance and Impact

• Scalability: By removing the dependency on external reward models and human annotations, Vision-SR1 offers a scalable path for improving VLMs without the prohibitive costs of data labeling or inference infrastructure.
• Addressing "Thinking over Seeing": The framework directly counters the trend in RL training where models learn to "think" (reason via language) without "seeing" (perceiving the image). It forces the model to prove it has seen the image by generating a sufficient description.
• Efficiency: The approach achieves state-of-the-art results with minimal computational overhead, making advanced visual reasoning accessible for models with limited resources.
• Future Direction: The paper highlights that accuracy gains in RL can sometimes be spurious (exploiting shortcuts); Vision-SR1 provides a mechanism to disentangle true visual grounding from language shortcut learning, setting a new standard for evaluating VLM reasoning.