Technical Summary: MIRL – Mutual Information-Guided Reinforcement Learning for Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) frequently exhibit visual perception errors and hallucinations, where generated text contradicts visual facts. These errors compromise accuracy in complex reasoning tasks. While Reinforcement Learning with Verifiable Rewards (RLVR) offers a solution by optimizing policies based on answer correctness, prevailing methods face two critical limitations:

1. Inefficient Exploration: Standard RLVR methods allocate sampling budgets uniformly across all trajectories. However, if the initial visual description contains critical errors (e.g., misidentified objects or missing details), the subsequent reasoning is "doomed to fail." Generating full trajectories for these low-quality descriptions wastes significant computational resources.
2. Reward Blindness: Existing methods typically rely on sparse, sequence-level rewards (final answer correctness). When a trajectory fails, the reward signal cannot distinguish whether the error originated from the visual perception stage, the reasoning logic, or the answer formulation. This ambiguity hinders targeted optimization of the visual grounding stage.

2. Methodology: MIRL

The authors propose MIRL (Mutual Information-guided Reinforcement Learning), a decoupled framework that addresses sampling inefficiency and reward blindness by leveraging Mutual Information (MI) between generated descriptions and visual inputs.

Core Mechanisms

A. MI as a Pre-Screening Signal (Sampling Efficiency)
The method posits that the quality of a visual description can predict downstream task success. It utilizes MI to measure the dependency of generated description tokens on the visual input:
$$MI(D; I | q) = \frac{1}{k} \sum_{t=1}^{k} [H(y_t | q, y_{<t}) - H(y_t | I, q, y_{<t})]$$
High MI indicates the model is generating content dependent on the image, while low MI suggests reliance on textual priors (hallucination).

B. Sample-Evaluate-Fork Strategy
Instead of uniformly sampling $G$ complete trajectories, MIRL employs a three-phase strategy:

1. Pre-Sampling: Generate $N$ candidate responses, extracting only the initial description segment $D$.
2. MI-Based Selection: Compute MI scores for all $N$ descriptions. Select the top-$K$ high-MI candidates.
3. Forking: For the selected $K$ descriptions, generate $M$ additional reasoning branches (forks) to create a tree structure. Low-MI descriptions are discarded from full trajectory generation but retained for perception training.
   This concentrates the computational budget on high-potential trajectories.

C. Decoupled Reward Assignment
To resolve reward blindness, MIRL decouples the training signals:

• MI-Based Reward: All $N$ pre-sampled descriptions receive an immediate reward based on their MI score. This provides dense, low-latency feedback specifically for optimizing visual perception and format alignment.
• Task-Based Reward: Only the $K(M+1)$ complete trajectories from selected descriptions receive the standard verifiable reward (answer correctness) to supervise reasoning and answer formulation.
• Decoupled Updates: Policy gradients are computed separately for description tokens (using MI advantages) and full trajectories (using task advantages), ensuring visual perception optimization is not confounded by reasoning quality.

3. Key Contributions

• MI-Guided Budget Allocation: The paper demonstrates that MI serves as a computationally cheap and reliable proxy for visual perception quality, enabling intelligent resource allocation via a "forking" mechanism that reduces the number of complete trajectories required.
• Decoupled Training Framework: MIRL introduces a novel training paradigm that separates visual perception optimization from reasoning optimization, providing independent signals to resolve the credit assignment problem inherent in sparse RLVR.
• Modular Integration: The approach is designed to be compatible with existing RL algorithms (e.g., GRPO, DAPO) by modifying only the sampling strategy and reward computation, without restructuring the core optimization framework.

4. Experimental Results

Experiments were conducted on six benchmarks spanning mathematical reasoning (MathVista, MathVerse, We-Math), multi-modal chain-of-thought (M3CoT), and general visual understanding (MMStar, RealWorldQA) using the Qwen2.5-VL-7B-Instruct model.

• Performance vs. Cost: MIRL-12 (pre-sampling 10, selecting top-6, forking 1) achieved an average accuracy of 70.22%, surpassing the DAPO-16 baseline (69.76%) while using 25% fewer complete trajectories (12 vs. 16).
• Efficiency Gains: MIRL-8 achieved 68.38% accuracy, comparable to DAPO-12 (69.17%) but with 33% fewer trajectories.
• Ablation Studies:
  • Selection Strategy: Replacing MI-based selection with random selection or selecting low-MI descriptions resulted in significant performance drops, confirming that MI-guided selection is the primary driver of efficiency.
  • Reward Configuration: Using MI rewards alone or task rewards alone yielded suboptimal results. The decoupled approach (combining both) achieved the best balance, validating the necessity of independent supervision for perception and reasoning.
• Generalization: The correlation between MI scores and task accuracy was consistent across diverse domains (Pearson $r = 0.71$ to $0.94$), suggesting MI is a universal indicator of visual grounding quality.

5. Significance and Claims

The paper claims that MIRL reveals the significant value of mutual information as an intermediate evaluation signal in VLM reinforcement learning. By treating visual perception as a distinct bottleneck that can be screened and optimized independently, MIRL offers a path toward more efficient large-scale VLM training.

The authors emphasize that their work opens new research directions for efficient large-scale VLM training by demonstrating that:

1. Visual perception errors can be detected early and filtered out before expensive reasoning generation.
2. Decoupling perception and reasoning rewards allows for targeted optimization of visual grounding without sacrificing reasoning capabilities.

The paper maintains a modest stance regarding limitations, noting that the current implementation focuses on static images and mathematical/scientific reasoning, and that further validation is needed for dynamic visual content (video) and diverse model architectures.