DynaWeb: Model-Based Reinforcement Learning of Web Agents

1. Problem Statement

The development of autonomous web agents powered by Large Language Models (LLMs) faces a critical bottleneck: the cost, inefficiency, and risk of online Reinforcement Learning (RL) in live web environments.

• Challenges: Training agents via direct interaction with the live internet requires massive computational resources, incurs financial costs (e.g., API fees, server usage), and carries significant risks (e.g., unintended purchases, account lockouts, data corruption).
• Limitations of Current Approaches:
  • Pure Online RL: Too expensive and hazardous for large-scale policy optimization.
  • World Models as Inference Tools: Existing methods use world models only for short-horizon lookahead or action evaluation at inference time, not as a core training substrate.
  • Offline RL/SFT: These methods often lack the adaptability and robustness gained from interactive trial-and-error learning.
• Core Question: How can agents retain the benefits of online RL (robustness, exploration) while drastically reducing reliance on direct, risky interaction with the live web?

2. Methodology: The DynaWeb Framework

DynaWeb introduces a Model-Based Reinforcement Learning (MBRL) framework that trains web agents through "imagination" using a learned Web World Model (WWM). The framework is inspired by the Dyna architecture and Dreamer but adapted for the complex, structured nature of web navigation.

A. The Web World Model (WWM)

Instead of learning latent dynamics, the WWM operates directly in the observation space (structured accessibility trees).

• Input: Current accessibility tree ($o_t$), executed action ($a_t$), and task instructions ($I$).
• Output: A prediction of the next state.
• Novel Prediction Strategy: To avoid low-information gain from predicting entire pages (which often remain static), the model is trained to predict state change descriptions ($\Delta$) in natural language. During inference, the model generates $\Delta$ and applies it to the current state to construct the next accessibility tree ($o_{t+1}$).
• Training Data: Trained on curated trajectories from the NNetNav dataset, using a reasoning model (GPT-oss-120b) to generate ground-truth reasoning traces and state changes.

B. The Training Loop (Imagination-Driven RL)

DynaWeb treats the WWM as a synthetic web environment.

1. Imagined Rollouts: The agent policy ($\pi_\theta$) interacts with the WWM to generate multi-step trajectories ($\hat{\tau}$) without touching the live web.
2. Reward Signal: Upon trajectory termination, a model-based self-assessment evaluates task completion, providing a scalar reward ($\hat{r} \in \{0, 1\}$).
3. Hybrid Experience Interleaving: To prevent the agent from overfitting to WWM hallucinations, DynaWeb randomly interleaves real expert trajectories (ground-truth data from training sets) with the imagined rollouts during training.
   • Ratio: Experiments show a mix of roughly 50% imagined and 50% real data (specifically, 40% real data was found optimal for regularization).

C. Optimization Algorithm

The framework utilizes Group Sequence Policy Optimization (GSPO).

• Unlike standard token-level PPO, GSPO assigns a single sequence-level importance ratio to the entire trajectory.
• This allows for stable credit assignment over long-horizon tasks with sparse terminal rewards, treating the whole rollout as a single unit of experience.

3. Key Contributions

1. First Online MBRL for Web Agents: DynaWeb is the first framework to elevate the web world model from a planning tool to a core component of online policy optimization, enabling agents to "dream" their training data.
2. Structured State Prediction: Introduces a method for predicting web state transitions via natural language state-change descriptions ($\Delta$) rather than raw text generation, improving simulation fidelity.
3. Hybrid Training Strategy: Demonstrates that interleaving real expert trajectories with imagined rollouts stabilizes learning, correcting systematic model biases while maintaining the efficiency of simulation.
4. Scalable RL Pipeline: Provides a viable path to scale up online agentic RL without the prohibitive costs and risks of live internet interaction.

4. Experimental Results

The authors evaluated DynaWeb on two challenging benchmarks: WebArena (simulated environments) and WebVoyager (live websites).

• Performance Gains:
  • WebArena: DynaWeb achieved an average Success Rate (SR) of 31.0%, outperforming the previous state-of-the-art Offline-RL baseline (WebRL, 26.7%) by a relative margin of 16.1%. It consistently beat baselines across diverse domains (Reddit, GitLab, CMS, Shopping).
  • WebVoyager: DynaWeb achieved the highest overall SR (38.7%), outperforming proprietary models like GPT-4o (31.1%) and other open-source RL agents.
• Ablation Studies:
  • Dream Length: Optimal performance occurred with a dream length of 4–5 steps. Longer rollouts suffered from compounding hallucinations, while shorter ones lacked sufficient depth for complex tasks.
  • Real Data Ratio: Adding ~40% real expert data significantly boosted performance over pure simulation or pure SFT, acting as a critical regularizer.
  • Necessity of Training: A frozen general-purpose LLM used as a world model performed significantly worse than the explicitly trained DynaWeb WWM, proving that environment-specific transition dynamics must be learned.

5. Significance and Impact

• Paradigm Shift: DynaWeb shifts the paradigm from "learning by doing on the live web" to "learning by imagining in a synthetic world," making large-scale RL for web agents feasible and safe.
• Safety and Cost: It eliminates the risk of irreversible actions (e.g., accidental purchases) during the training phase and removes the dependency on expensive live web infrastructure for data collection.
• Future Direction: The paper establishes that imagination-driven training is a scalable solution for long-horizon decision-making in complex, dynamic environments. It highlights that while current world models struggle with very long-horizon planning, the combination of simulation and real-world regularization offers a robust path forward for general-purpose AI assistants.

In summary, DynaWeb successfully demonstrates that web agents can be trained more effectively and safely by leveraging a learned world model to generate synthetic experience, augmented by real expert data, rather than relying solely on costly and risky interactions with the live internet.