Technical Summary: Vector Policy Optimization (VPO)

Problem Statement

Current Large Language Model (LLM) post-training paradigms, particularly those utilizing Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO), optimize for a single, pre-specified scalar reward. While effective for converging on a specific high-probability response, this approach often leads to "mode collapse," where the policy produces low-entropy response distributions. This lack of diversity becomes a critical bottleneck when LLMs are deployed within test-time search procedures (e.g., rejection sampling, evolutionary algorithms like AlphaEvolve, or best-of-$k$ selection).

In search-augmented systems, the training phase should ideally provide a rich pool of diverse, competent candidates for the search algorithm to exploit. However, standard scalar RL training collapses the candidate distribution toward a single optimal trade-off, causing additional samples to become near-duplicates. Consequently, the search procedure cannot discover superior solutions that require different trade-offs between competing objectives (e.g., code correctness vs. efficiency, or different reasoning paths in multi-hop questions).

Methodology: Vector Policy Optimization (VPO)

The authors propose Vector Policy Optimization (VPO), an RL algorithm designed to explicitly train policies to produce diverse sets of solutions that span the Pareto frontier of vector-valued rewards. VPO operates on the premise that in many realistic tasks (code generation, multi-hop reasoning, tool use), rewards naturally decompose into vector components (e.g., per-test-case correctness, per-criterion ratings).

VPO consists of two core components:

1. Multi-Answer Chains as In-Context Exploration:
   Instead of generating a single response per prompt, the model is trained to emit a set of $m$ candidate completions ($S = \{y_1, \dots, y_m\}$) within a single autoregressive rollout. Each subsequent candidate can attend to the previously generated answers in the context, allowing the model to explicitly recognize which regions of the solution space are covered and steer new candidates toward unexplored areas. This transforms diversity from a byproduct of sampling noise into an explicit, in-context mechanism.

2. Set-Level Optimization via Stochastic Scalarization:
   To incentivize the model to produce diverse sets rather than collapsing to a single mode, VPO replaces the fixed scalar reward with a distribution over reward weightings.

   • For each rollout, a weight vector $w$ is sampled from a Dirichlet distribution ($w \sim \text{Dir}(\alpha)$, where $\alpha=1$ implies a uniform distribution over the simplex).
   • The reward for the set $S$ is calculated as the expected maximum scalarized reward across the sampled weights:
     $$R(S) = \mathbb{E}_{w \sim \text{Dir}(\alpha)} \left[ \max_{y \in S} w^\top r(x, y) \right]$$
   • This objective rewards the model for covering the Pareto frontier: different elements in the set $S$ are incentivized to specialize in different trade-offs of the reward components.

VPO is implemented as a drop-in replacement for the GRPO advantage estimator. It uses the GRPO framework but calculates advantages based on the set-level reward derived from stochastic scalarizations, sharing the same $K$ weight vectors across a group of $G$ rollouts to ensure comparability.

Key Contributions

1. Reframing RL Post-Training: The paper argues that in systems with test-time search, the role of RL training should shift from converging on a single best response to maximizing the diversity of a set of competent solutions, leaving exploitation to the search procedure.
2. Vector Reward Structure: It demonstrates that the vector-valued nature of rewards in practical tasks provides a natural mechanism for diversity. By training the model to cover the Pareto frontier of different objectives, the policy maintains a population of alternatives that are optimal under different weightings.
3. VPO Algorithm: The authors propose VPO, which combines multi-answer generation with randomized reward scalarizations. This allows the model to generate diverse candidate sets within a single rollout, optimizing for coverage of the reward space rather than a single scalar peak.

Experimental Results

The authors evaluated VPO across four diverse domains: Maze (navigation with item trade-offs), MuSiQue (multi-hop QA), EUREQA (chain reasoning), and ToolRL (function calling). They compared VPO against several baselines, including standard GRPO, Multi-RLVR (multi-answer with fixed scalar reward), Random-Weighting GRPO, and Max-at-$k$ training.

• Test-Time Scaling: Across all domains, VPO consistently outperformed scalar baselines on best@k metrics. While scalar GRPO performance plateaued quickly as $k$ increased (indicating candidate collapse), VPO continued to improve with additional samples, showing a widening gap as the search budget grew.
• Diversity Metrics: VPO maintained high reward-space diversity (measured by pairwise L1 distance between reward vectors), whereas baselines like Multi-RLVR and standard GRPO collapsed to near-identical reward vectors early in training.
• Hard Problems & Evolutionary Search: In a case study on LiveCodeBench (a competitive programming benchmark), a VPO-trained model (Qwen2.5-Coder-7B) significantly outperformed a matched-compute GRPO model. Crucially, when integrated into OpenEvolve (an evolutionary search loop), VPO models solved problems that GRPO models could not solve at any candidate budget, demonstrating that VPO unlocks solutions requiring complex trade-offs.
• Ablation Studies:
  • Multi-answer generation alone (without stochastic scalarization) was insufficient to maintain diversity.
  • Stochastic scalarization alone (without multi-answer chains) did not yield the same gains.
  • Goal-conditioned GRPO (conditioning on specific weights) failed to match VPO, suggesting that generating a diverse set in-context is more effective than explicitly conditioning on weights at inference time.
  • The benefits of VPO were most pronounced when reward components were non-collinear (i.e., when true trade-offs existed). In domains with near-collinear rewards (e.g., UltraFeedback), VPO's advantage diminished, aligning with the theoretical premise.

Significance and Claims

The paper posits that as test-time search becomes a standard component of AI systems, optimizing for diversity must become the default post-training objective. The authors claim that VPO provides a principled way to separate exploration (training for a diverse pool) from exploitation (search-time selection).

The significance of VPO lies in its ability to:

• Unlock Harder Problems: By preserving diverse strategies that may appear locally suboptimal under a fixed scalar but contain partial reasoning patterns useful for search, VPO enables the discovery of solutions that scalar-trained policies miss entirely.
• Scale with Compute: The performance gap between VPO and scalar baselines widens as the test-time search budget increases, making VPO particularly valuable for compute-intensive inference scenarios.
• Drop-in Compatibility: VPO requires no architectural changes to existing LLMs and functions as a simple replacement for the advantage estimator in standard RL frameworks like GRPO.

The authors conclude that while VPO sacrifices pass@1 performance (as it prioritizes exploration over immediate exploitation), it is the superior approach for regimes where test-time search is available, effectively turning policy optimization into a coverage problem over the Pareto front.