Technical Summary: Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning

Problem Statement

Large Vision-Language Models (VLMs) are increasingly deployed as agents in interactive, multi-turn environments where success depends on coherent reasoning and sequential decision-making. However, current VLMs suffer from "nearsightedness," often reasoning based solely on the most recent observation while neglecting the context of past actions and feedback. This leads to repetitive, non-adaptive behaviors where multi-turn decision-making devolves into disjointed single-turn generations.

Existing reinforcement learning (RL) approaches for VLMs generally adopt one of two modeling paradigms, both with limitations:

1. Token-wise RL: Treats each generated token as an individual action. While this provides fine-grained guidance for language generation, it often fails to leverage intermediate turn-level feedback effectively.
2. Turn-wise RL: Treats the entire sequence of tokens generated in a turn as a single macro-action. This better utilizes intermediate feedback for intra-turn differentiation but lacks the fine-grained control necessary for stable autoregressive generation.

Prior works attempting to combine these levels often rely on complex hierarchical models or heuristic credit assignment without a unified theoretical foundation, leading to suboptimal bias-variance trade-offs.

Methodology: HyGAE

The authors propose HyGAE (Hybrid Advantage Estimation with Unified Critic), an actor-critic framework that jointly optimizes token-level and turn-level objectives. The method is grounded in a theoretical unification of the two RL formulations.

Theoretical Formulation

The paper establishes that token-wise and turn-wise RL can be viewed under a unified Partially Observable Markov Decision Process (POMDP) framework.

• Equivalence of Gradients: The authors prove that under specific discount factor constraints ($\gamma_t = \gamma^{I_t}$, where $I_t$ is the turn length), the policy gradients for token-wise and turn-wise objectives are identical.
• Unified Critic: They demonstrate that a single value function (critic) can estimate values for both levels. Specifically, the turn-wise value at the start of a turn is equivalent to the token-wise value at the final token of that turn, provided the discount factors are aligned. This eliminates the need for separate value models, reducing computational overhead.

Hybrid Advantage Estimation

Instead of choosing between token-wise or turn-wise advantages, HyGAE employs a hybrid advantage function:
$$\hat{A}^i_t = \alpha \hat{A}_t + (1 - \alpha) \hat{A}^i_t$$
Where $\hat{A}_t$ is the turn-wise advantage and $\hat{A}^i_t$ is the token-wise advantage.

• Turn-wise Advantage: Computed using Generalized Advantage Estimation (GAE) over turns, preserving the turn-level reward signal and avoiding excessive discounting through intermediate reasoning tokens.
• Token-wise Advantage: Computed using GAE over tokens, providing fine-grained differentiation within a turn.
• Hybrid Return: To update the unified critic, the authors propose a specific formulation for the hybrid return. They broadcast the turn-wise return back to intermediate token positions by summing discounted rewards from the current token to the end of the turn, then combine this with the token-wise return. This precise analytic form is shown to be critical for stabilizing value function training.

Algorithm

HyGAE operates within a standard PPO (Proximal Policy Optimization) loop:

1. Collect trajectories involving multiple turns of interaction.
2. Compute turn-wise and token-wise advantages and returns separately.
3. Combine them using the mixing coefficient $\alpha$ to form the hybrid advantage and hybrid return.
4. Update the policy (actor) and the unified value model (critic) using the hybrid signals.

Key Contributions

1. Theoretical Unification: The paper provides a formal formulation revealing the intrinsic similarity between token-wise and turn-wise RL, proving that their surrogates share the same gradient under specific conditions.
2. Hybrid Advantage Estimation: A novel method that linearly combines token-wise and turn-wise advantages. The authors prove this approach balances the bias-variance trade-off: turn-wise estimators have lower bias but higher variance, while token-wise estimators have higher bias but lower variance. The hybrid approach mitigates the weaknesses of both.
3. Unified Critic Training: A proof that a single value model can effectively estimate values for both granularities, significantly improving computational efficiency compared to hierarchical approaches.
4. Empirical Performance: The method achieves State-of-the-Art (SOTA) results across five diverse multi-turn decision-making benchmarks.

Experimental Results

The authors evaluated HyGAE on five benchmarks: Sokoban, FrozenLake, Navigation, Primitive Skill (robotic manipulation), and VIRL (real-world navigation).

• Performance: HyGAE achieved an average success rate of 91% across all tasks, significantly outperforming baselines.
  • It surpassed proprietary models (e.g., GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5) which averaged around 0.5–0.6.
  • It significantly outperformed frozen open-source VLMs (average < 0.1).
  • Among RL methods, it outperformed Token-PPO (0.81), Turn-PPO (0.47), and VAGEN-Base (0.74).
• Ablation Studies:
  • The mixing coefficient $\alpha = 0.5$ (equal weighting) yielded the best performance, confirming the value of combining both signals.
  • The specific formulation of the hybrid return (broadcasting turn rewards with discounting) was identified as crucial; alternative formulations led to training collapse.
  • HyGAE demonstrated significantly lower variance in success rates compared to standard PPO, indicating more stable optimization.
• Qualitative Analysis: Visualizations showed that while frozen models often repeat failed actions or static plans, HyGAE successfully adapts its strategy based on environmental feedback, demonstrating coherent multi-turn reasoning.

Significance

The paper claims that the primary bottleneck in VLM agentic training is the mismatch between single-turn training paradigms and the requirements of multi-turn interaction. By theoretically unifying token-wise and turn-wise optimization and introducing a computationally efficient hybrid advantage estimation, HyGAE bridges the gap between high-level strategic planning and low-level language generation. The work suggests that precise analytic forms for advantage and return estimation are critical for stable optimization in agentic RL, offering a robust framework for training VLMs to act as effective agents in complex, interactive environments.