All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models

1. Problem Statement

The paper addresses a critical limitation in current Reinforcement Learning (RL) approaches for Vision-Language Models (VLMs), specifically Group Relative Policy Optimization (GRPO). While RL has significantly enhanced the reasoning capabilities of VLMs, the authors identify a fundamental behavioral distinction between RL-trained models and their base counterparts:

• RL Models (Convergent): Tend to engage in "deep but narrow" reasoning. They excel at refining a single dominant strategy but suffer from diversity collapse, where they prematurely converge to a limited subset of reasoning paths, discarding alternative strategies. This leads to local optima and poor scalability when faced with complex problems requiring diverse approaches.
• Base Models (Divergent): While less refined along a single path, they exhibit "broad and diverse" thinking patterns. When given multiple sampling attempts, base models often succeed in solving problems that RL models fail, by leveraging alternative reasoning pathways (e.g., using exclusion strategies instead of sequential counting, or validation instead of equation solving).

The core problem is that standard RL training (GRPO) suppresses the inherent divergent thinking of base models, limiting the model's ability to explore the solution space effectively and scale performance at test time.

2. Methodology: Multi-Group Policy Optimization (MUPO)

To address diversity collapse, the authors propose MUPO, a drop-in replacement for GRPO designed to incentivize divergent thinking across multiple solutions.

Key Components of MUPO:

• Group Partitioning: Instead of treating all sampled responses as a single pool, MUPO partitions $N$ sampled responses into $K$ distinct groups based on their reasoning embeddings (using constrained clustering). Each group represents a distinct "mode" or reasoning strategy.
• Localized Advantage Estimation: Unlike GRPO, which computes advantages globally across all samples, MUPO calculates advantages locally within each group. This ensures that each reasoning mode is refined independently without being penalized by the performance of other modes.
• Diversity Reward: To prevent groups from collapsing into a single strategy, MUPO introduces a diversity reward ($R_{div}$). This reward is calculated based on the average cosine distance between a response's reasoning embedding and those of responses in other groups.
  • $R_{total} = R_{acc} + R_{fmt} + \lambda \cdot \mathbb{I}[R_{acc}=1] \cdot R_{div}$
  • The diversity reward is only applied to correct answers to prevent "reward hacking" (pursuing diversity at the cost of correctness).
• Annealing Schedule: The weight of the diversity reward ($\lambda$) follows a cosine annealing schedule. It starts high to encourage broad exploration in early training and gradually decreases to shift focus toward exploiting the most promising strategies in later stages.
• Load Balancing: A scaling factor ($w_k$) is applied to each group's objective to prevent larger groups from dominating the optimization process, ensuring balanced learning across all discovered strategies.

3. Key Contributions

1. Behavioral Analysis: The paper provides empirical evidence that RL models suffer from "diversity collapse," converging to narrow strategies, whereas base models maintain broader, more diverse reasoning patterns that are crucial for solving complex, multi-faceted problems.
2. MUPO Algorithm: The proposal of a novel policy optimization algorithm that explicitly balances exploration (divergent thinking across groups) and exploitation (refining individual strategies within groups).
3. State-of-the-Art Performance: Demonstration that MUPO achieves superior performance on established benchmarks compared to strong RL baselines (e.g., Vision-R1, R1-OneVision) and base models, particularly in test-time scaling scenarios (acc@4).

4. Experimental Results

The authors evaluated MUPO-Thinker (based on Qwen2.5-VL 3B and 7B) on nine reasoning benchmarks, including MathVerse, MathVista, LogicVista, and HallusionBench.

• Accuracy Improvements (acc@1):
  • 7B Model: Achieved a 2.5% average gain on mathematical benchmarks and 2.3% on general-purpose benchmarks over previous bests.
  • 3B Model: Surpassed same-scale baselines with gains of 2.0% (math) and 2.4% (general).
• Test-Time Scaling (acc@4):
  • MUPO-Thinker-7B showed significant improvements in multi-sampling scenarios, outperforming existing RL models by 6.0% on math benchmarks and 6.2% on general benchmarks.
  • The 3B model trained with MUPO achieved performance comparable to strong 7B baselines, highlighting the efficiency of the method.
• Qualitative Analysis:
  • Visualizations (t-SNE) confirmed that MUPO maintains a multimodal distribution of reasoning embeddings, whereas GRPO collapses into a single dense cluster.
  • Case studies showed MUPO successfully employing alternative strategies (e.g., using reference objects for height estimation) where GRPO failed due to rigid, error-prone sequential reasoning.

5. Significance and Impact

• Paradigm Shift in RL for VLMs: The paper challenges the prevailing notion that RL must strictly converge to a single optimal path. It argues that preserving divergent thinking is essential for robust reasoning, especially in visual tasks where multiple valid approaches exist.
• Scalability: By enabling models to explore multiple reasoning "roads" simultaneously, MUPO significantly enhances test-time scaling capabilities, allowing smaller models to outperform larger, convergent models.
• Generalizability: The approach is a "drop-in" replacement for GRPO, making it easily adoptable for other RL-based training pipelines in multimodal and language domains.
• Human-Aligned Reasoning: The method mimics human problem-solving, where individuals often approach a problem from multiple angles before settling on the best solution, rather than rigidly adhering to a single initial hypothesis.

In conclusion, MUPO successfully bridges the gap between the depth of RL and the breadth of base models, offering a robust framework for training Vision-Language Models that are both deep in reasoning and diverse in strategy.