OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks

1. Problem Statement

The paper addresses two critical challenges in applying Reinforcement Learning (RL), specifically Group Relative Policy Optimization (GRPO), to Multimodal Large Language Models (MLLMs) across diverse visual tasks:

• Extreme Variance in Reward Topologies: Visual tasks range from sparse, binary rewards (e.g., Math VQA, MCQs) to dense, continuous rewards (e.g., Intersection-over-Union in grounding tasks) and sparse regression signals (e.g., OCR). Standard GRPO relies on linear scaling (mean and variance normalization) within a group. This approach fails to guarantee inter-task gradient equity. It is highly susceptible to:
  • Heavy-tail outliers: A single anomalous reward can inflate variance, suppressing learning signals for normal responses.
  • Imbalanced updates: High-variance tasks dominate gradient updates, while low-variance tasks are suppressed.
  • Structural pathologies: Linear transformations preserve the original distribution shape (e.g., bimodal step functions), failing to smooth out structural issues inherent in diverse tasks.
• Balancing Perception vs. Reasoning: MLLMs often struggle to simultaneously maintain fine-grained visual perception (requiring concise, direct outputs) and multi-step reasoning (requiring extended chains of thought). Existing methods to enhance perception often require costly data annotations or additional modules, limiting scalability.

2. Methodology

The authors propose OpenVLThinkerV2, which integrates a novel RL objective and task-level shaping mechanisms to solve the above problems.

A. Gaussian GRPO (G2RPO)

Instead of linear standardization, G2RPO employs non-linear distributional matching using 1D Optimal Transport (OT).

• Core Mechanism: It maps the empirical reward distribution of any given task directly to a standard normal distribution, $N(0, 1)$.
• Implementation:
  1. Ranking: Rewards within a group are ranked to compute empirical probabilities ($p_i$).
  2. Quantile Mapping: These probabilities are mapped to the inverse Cumulative Distribution Function (CDF) of the standard normal distribution ($\Phi^{-1}$).
  3. Tie-Breaking: Identical rewards receive the average of their target quantiles to ensure consistent learning signals.
• Theoretical Benefits:
  • Outlier Robustness: Mathematically caps extreme rewards at the highest quantile, preventing gradient explosions.
  • Symmetry: Converts bimodal or skewed distributions into symmetric Gaussian tails, ensuring balanced updates for positive and negative rewards.
  • Inter-task Equity: By forcing all tasks to converge to $N(0, 1)$, it ensures uniform variance across diverse tasks, eliminating the need for complex momentum-based hyperparameters (like in EMA-GRPO).

B. Task-Level Shaping Mechanisms

To balance perception and reasoning, the authors introduce two dynamic shaping mechanisms:

1. Response Length Shaping:

   • Goal: Encourage extended reasoning for complex queries while enforcing concise outputs for vision-centric tasks to prevent hallucinations.
   • Method: A trapezoidal reward envelope is applied. It rewards generation lengths falling within an optimal range ($[L_{low}, L_{high}]$) and penalizes lengths that are too short or too long.
   • Effect: Dynamically elicits "Chain of Thought" for math/science tasks while reducing "overthinking" in OCR or grounding tasks.

2. Entropy Shaping:

   • Goal: Prevent entropy collapse (premature over-reliance on high-probability tokens in vision tasks) and entropy explosion (generating incoherent text in complex reasoning tasks).
   • Method: A regularization loss ($L_{ent\_reg}$) is added that penalizes entropy if it falls outside a task-specific optimal interval $[H_{min}, H_{max}]$.
   • Effect: Confines the model's exploration zone, ensuring stable convergence across both reasoning-heavy and perception-heavy domains.

3. Key Contributions

1. G2RPO Objective: A novel RL training objective that replaces linear scaling with non-linear distributional matching (via 1D Optimal Transport) to ensure inter-task gradient equity and robustness to outliers.
2. Task-Level Shaping: Introduction of response length and entropy shaping mechanisms that dynamically balance fine-grained perception and multi-step reasoning without requiring extra modules or costly annotations.
3. OpenVLThinkerV2: A robust, general-purpose multimodal model that establishes new State-of-the-Art (SOTA) results among open-source models, outperforming leading proprietary frontier models (e.g., GPT-5, Gemini 2.5 Pro) in specific domains.

4. Experimental Results

The model was evaluated on 18 diverse benchmarks across six categories: General VQA, Math VQA, Chart VQA, Spatial Reasoning, Document Understanding, and Visual Grounding.

• General & Math Reasoning:
  • MMMU: 71.6% (Surpassing GPT-4o).
  • MathVista: 79.5% (Significant margin over GPT-4o).
  • MMBench: 88.2%.
• Document Understanding:
  • OCRBench: 911 (Outperforming specialized models like DeepEyesV2 and proprietary models GPT-5 and Gemini 2.5 Pro).
  • DocVQA: 96.7%.
• Spatial Reasoning:
  • EmbSpatial: 83.1% (Highest performance, outperforming GPT-5).
  • RoboSpatial: 63.2% (Outperforming GPT-5 and Gemini 2.5 Pro).
• Visual Grounding:
  • RefCOCO/+/g: Achieved 93.4%/88.2%/90.4%, surpassing specialized grounding experts like Grounding DINO.
• Ablation Studies:
  • G2RPO provided the most significant initial boost.
  • Adding entropy shaping improved reasoning tasks.
  • Adding length shaping provided broad improvements across all domains.
  • The combination of all components yielded the highest performance.

5. Significance

• Scalability: The proposed method avoids the computational overhead and data annotation costs of previous perception-enhancement techniques, making it highly scalable for multi-domain training.
• Theoretical Advancement: By utilizing Optimal Transport to enforce a Gaussian topology, the paper provides a mathematically rigorous solution to the "reward heterogeneity" problem in multi-task RL, offering a blueprint for stable training in highly diverse environments.
• Generalist Capability: OpenVLThinkerV2 demonstrates that a single model can effectively master both "fast" visual perception and "slow" logical reasoning, challenging the notion that these capabilities require separate, specialized architectures.
• Open Source Leadership: The model sets a new benchmark for open-source MLLMs, proving they can compete with and exceed the capabilities of leading proprietary models in complex visual reasoning and grounding tasks.