Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models

1. Problem Statement

The paper addresses a critical structural bottleneck in applying Reinforcement Learning from Verifiable Rewards (RLVR) to Large Vision-Language Models (LVLMs).

• The Core Issue: While RLVR (e.g., GRPO) has significantly improved reasoning in text-only LLMs, its application to LVLMs is hindered by a mismatch between dense visual token representations and sparse, structured visual signals (e.g., geometric diagrams, charts).
• The Bottleneck: Existing methods assume that dense tokenization suffices for visual encoding. However, this overlooks the fact that visual evidence in reasoning tasks is often spatially sparse (e.g., specific lines, angles, or symbols) while large portions of the image are semantically redundant (background).
• Consequence: This misalignment causes Visual Perception (VP) errors to become the primary failure mode in multimodal reasoning. Empirical analysis shows VP errors account for 48.9% of failures in models like Qwen2.5-VL, far exceeding calculation or rule application errors. Current reward mechanisms fail to couple visual representations with the RL optimization process, leading to suboptimal credit assignment.

2. Methodology: KAWHI

The authors propose KAWHI (Key-Region Aligned Weighted Harmonic Incentive), a plug-and-play reward reweighting mechanism designed to integrate structured visual information into uniform reward policy optimization (compatible with GRPO, GSPO, etc.). The method consists of four key stages:

A. Structure-Guided Union-Find (SGUF) for Visual Token Selection

To address information sparsity, KAWHI adaptively identifies semantically salient regions rather than processing all visual tokens equally.

• Feature Extraction: Converts images to CIELab space and computes a Structure Tensor based on gradient fields to capture local geometry (edges, strokes, symbols).
• Hierarchical Aggregation: Uses a Union-Find algorithm to group image patches based on structural coherence (eigenvalues of the tensor) and luminance similarity.
• Adaptive Sampling: Identifies "Key Regions" (high structural energy) and "Background Regions." It retains all tokens from key regions while sparsely sampling background regions, effectively compressing redundant visual data.

B. Vision-Critical Head Identification

Not all attention heads contribute equally to visual reasoning.

• Ablation Analysis: The authors perform global head ablation on the MME benchmark to identify vision-critical attention heads. Heads whose removal causes a significant drop in visual performance are selected as the "critical subset."
• Purpose: This ensures that subsequent credit assignment focuses only on the computational units responsible for visual perception.

C. Spatially-Aware Credit Assignment

KAWHI quantifies the alignment between the generated reasoning steps and the critical visual regions.

• Mechanism: It treats response tokens as Query (Q) vectors (information probes) and visual tokens from SGUF-selected key regions as Key (K) vectors (semantic identifiers).
• Alignment Score: It calculates the cosine similarity between Q and K vectors, restricted to the vision-critical heads. This generates a spatial saliency score ($\alpha_t$) for each token, indicating how much the model's reasoning relies on specific visual evidence.

D. Semantic-Preserving Spatial Weighting

To avoid fragmenting the reasoning chain, KAWHI applies weights at the paragraph level rather than the token level.

• Aggregation: Spatial saliency scores are aggregated via mean pooling within each paragraph (separated by \n\n).
• Reweighting: These paragraph-level scores are normalized into weights ($w_j$) using a temperature-scaled softmax.
• Integration: The standard group-relative advantage ($A_g$) in RLVR is modulated by these weights ($\hat{A}_{g,t} = A_{g,t} \cdot w_j$). This amplifies gradient signals for paragraphs containing critical visual evidence while suppressing noise from redundant segments.

3. Key Contributions

1. Structural Bottleneck Identification: The paper systematically identifies that the lack of explicit coupling between visual geometry and RL optimization is the primary cause of reasoning failures in LVLMs, with VP errors being the dominant failure mode.
2. KAWHI Framework: Introduces a novel, plug-and-play reward reweighting mechanism that:
   • Uses SGUF to filter visual tokens based on geometric structure.
   • Identifies vision-critical attention heads for precise attribution.
   • Implements paragraph-level credit reallocation to align spatial visual evidence with semantically decisive reasoning steps.
3. Generalizability: Demonstrates that KAWHI is compatible with various uniform reward optimization methods (GRPO, DAPO, GSPO) and does not require architectural modifications to the underlying model.

4. Experimental Results

Extensive evaluations were conducted on Qwen2.5-VL-7B and Qwen3-VL-4B across mathematical reasoning (MathVista, MathVerse, MathVision, WeMath) and chart understanding benchmarks.

• Performance Gains:
  • Mathematical Reasoning: Integrating KAWHI with GRPO on Qwen2.5-VL-7B improved the average score from 47.68 to 51.00 (+3.32 points). When combined with GSPO, the gain was even more significant, reaching 52.94 (+5.26 points).
  • Chart Understanding: On ChartQA and related benchmarks, KAWHI provided consistent gains (e.g., +4.9% on ChartMimic), proving its effectiveness in structured visual tasks.
  • Comparison: KAWHI outperformed existing fine-grained reward methods like VPPO and Step-GRPO, as well as other uniform reward baselines.
• Ablation Studies:
  • Removing SGUF (using random or inverse selection) significantly degraded performance, confirming the necessity of structure-guided token selection.
  • Using paragraph-level weighting was superior to token-level or sentence-level weighting, highlighting the importance of preserving semantic coherence.
  • Restricting calculations to vision-critical heads improved results, validating the head selection strategy.
• Efficiency: The method introduces a marginal computational overhead of only 3.6% (approx. 19.6s per step) while delivering substantial performance improvements.

5. Significance

• Theoretical Advancement: KAWHI bridges the gap between visual perception and reinforcement learning by explicitly modeling the spatial sparsity of visual information. It moves beyond treating images as a flat sequence of tokens, instead leveraging geometric priors to guide the RL optimization process.
• Practical Impact: By reducing hallucinations and improving the model's ability to focus on relevant visual cues (e.g., specific angles in geometry problems), KAWHI offers a scalable solution for enhancing multimodal reasoning without retraining the entire model architecture.
• Future Direction: The work suggests that future RLVR methods for LVLMs must incorporate structured visual alignment and spatial credit assignment to fully unlock the reasoning potential of multimodal models.