Technical Summary: GROW – Aligning GRPO with State-Action Modeling for Open-World VLM Agents

1. Problem Statement

Vision-Language Model (VLM) agents have demonstrated potential in open-world tasks (e.g., Minecraft), which require multi-turn visual perception and action execution. While Supervised Fine-Tuning (SFT) on expert demonstrations is the dominant training paradigm, it suffers from high data collection costs and limited scalability. Furthermore, SFT often lags behind Reinforcement Learning (RL) methods in performance.

Although Group Relative Policy Optimization (GRPO) has emerged as an effective RL algorithm for foundation models—eliminating the need for a separate value model by computing advantages across groups of sampled outputs—directly applying standard GRPO to open-world tasks presents significant challenges:

• Context Length: Standard GRPO treats a full interaction trajectory as a single training sample. In open-world environments, these trajectories accumulate excessive context lengths, often exceeding token limits.
• Noise: Full trajectories contain substantial irrelevant information and noise, degrading the quality of policy gradient estimation.
• Theoretical Mismatch: Standard GRPO assumes grouped samples are different responses to the same prompt. In open-world tasks, decomposing trajectories means grouped samples are conditioned on different local states, violating this assumption.

2. Methodology: The GROW Framework

The authors propose GROW (GRPO for Open-World VLM agents), a framework that adapts GRPO to open-world settings by decomposing trajectories into fine-grained state-action samples.

2.1 Trajectory Decomposition

Instead of optimizing full trajectories, GROW decomposes collected rollout trajectories $\tau = \{(s_1, a_1), \dots, (s_H, a_H)\}$ into individual state-action pairs.

• Reward Propagation: Since rewards in these environments are sparse and verifiable only at task completion ($R(\tau)=1$ for success, $0$ otherwise), the reward is propagated backward along the trajectory using a discount factor $\gamma$:
  $$r_{i,t} = \gamma^{H_i - t} R(\tau_i)$$
  This ensures state-action samples closer to the successful outcome receive stronger learning signals.

2.2 State-Action Advantage Computation

Within a rollout group of $G$ parallel environments, GROW computes relative advantages over the decomposed state-action samples rather than full trajectories.

• The advantage $A_{i,t}$ is calculated by normalizing the discounted rewards $r_{i,t}$ across the group:
  $$A_{i,t} = \frac{r_{i,t} - \mu}{\sigma}$$
  where $\mu$ and $\sigma$ are the mean and standard deviation of the reward set within the group.
• The training objective follows the standard PPO-style clipped objective but operates on these state-action units:
  $$J = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{H_i} \sum_{t=1}^{H_i} \min(\rho_{i,t}(\theta) A_{i,t}, \text{clip}(\rho_{i,t}(\theta), 1-\epsilon, 1+\epsilon) A_{i,t}) \right]$$

2.3 Surrogate Analysis

The authors address the theoretical concern that grouped samples in GROW are conditioned on different local states ($s_{i,t}$) rather than an identical prompt. Under simplifying assumptions (uniform trajectory length $H$ and consistent interaction steps), they derive a surrogate analysis showing that the objective decomposes into two terms:

1. $J_{traj}$: Preserves the trajectory-level relative preference by comparing trajectory rewards $R(\tau_i)$ against the group average $S$.
2. $J_{step}$: Captures temporal discounting effects, refining credit assignment within successful trajectories to emphasize decisions directly related to final success.
   The analysis concludes that despite the departure from the identical-prompt assumption, the objective retains a valid and effective relative policy optimization signal.

3. Key Contributions

1. Framework Proposal: Introduction of GROW, a RL framework that combines cold-start training on state-action samples with GRPO-based policy refinement via trajectory decomposition.
2. Theoretical Justification: Provision of a surrogate analysis demonstrating that relative policy optimization remains effective even when grouped samples are conditioned on different local states, provided reasonable tractability approximations hold.
3. Empirical Validation: Extensive testing on over 800 Minecraft tasks, demonstrating that the framework achieves State-of-the-Art (SOTA) performance in both success rate and execution efficiency.

4. Experimental Results

Experiments were conducted in Minecraft (Java Edition) using the MCU benchmark, covering three categories: Embodied tasks (navigation/mining), GUI tasks (crafting/smelt), and Combat tasks. The base model was Qwen2-VL-7B-Instruct.

• Performance Gains: GROW achieved SOTA results across all categories:
  • Embodied: ASR improved from 50.4% (previous best) to 59.6%.
  • GUI: ASR improved from 39.1% to 68.4%.
  • Combat: ASR improved from 39.2% to 49.0%.
• Efficiency: The average number of steps required to complete tasks was significantly reduced (e.g., from 263 to 128 steps in embodied tasks).
• Generalization: GROW showed strong generalization to unseen tasks (RL-unseen), improving success rates significantly over the initialized policy without overfitting to the training set.
• Ablation Studies:
  • vs. PPO: GROW outperformed PPO in both success rates and convergence speed.
  • Discount Factor ($\gamma$): A value of $\gamma=0.995$ yielded the best balance between long-horizon credit assignment and local efficiency.
• Behavioral Analysis: Diagnostic tasks revealed that GROW improved specific skills such as stable target fixation, active target reacquisition (searching for lost targets), and distractor-robust GUI operation.

5. Significance and Claims

The paper claims that GROW establishes a new SOTA for open-world VLM agents by effectively bridging the gap between advanced RL algorithms (GRPO) and the specific constraints of open-world environments (long, noisy trajectories).

• Core Achievement: The framework successfully mitigates the issues of excessive context accumulation and noise inherent in full-trajectory RL while preserving the core relative optimization signal of GRPO.
• Skill Acquisition: The results suggest that GROW fosters reusable interaction strategies (e.g., active searching, robust GUI handling) rather than mere trajectory memorization.
• Scalability: By decomposing trajectories, the method offers a scalable path for training VLM agents in complex, multi-turn environments where value estimation is costly and unstable.

The authors note limitations, including the current lack of a memory module for very long-horizon tasks and the reliance on specific simulators (Minecraft), suggesting future work could explore summarizing history and expanding to diverse open-world environments.