R-WoM: Retrieval-augmented World Model For Computer-use Agents

This paper proposes R-WoM, a retrieval-augmented world model that mitigates the hallucination and long-horizon planning limitations of Large Language Models in computer-use agents by integrating factual external knowledge, thereby significantly improving performance in state prediction and task execution.

The Gist

Here is an explanation of the paper "R-WOM: Retrieval-Augmented World Model for Computer-Use Agents" using simple language and creative analogies.

The Big Picture: The "Daydreaming" Problem

Imagine you hire a very smart, well-read assistant (an AI Agent) to help you do tasks on your computer, like "Download this file and email it to my boss."

In the past, these assistants tried to figure out how to do this by daydreaming. They would close their eyes, imagine the future steps, and guess what would happen if they clicked "Save" or "Send." This is called a World Model.

• The Good News: They are great at guessing the next step. If you click "Save," they know the file will appear on the desktop.
• The Bad News: They are terrible at guessing the whole journey. If the task is long and complicated, their daydreams start to get fuzzy. They might hallucinate (make things up), forget where the cursor is, or suggest steps that look logical but are actually impossible to do in the real software. It's like trying to navigate a new city using only a map from 10 years ago; you might get lost because the streets have changed.

The Solution: R-WoM (The "Google Maps" Approach)

The authors of this paper realized that instead of relying on the AI's internal memory (which is outdated and prone to daydreaming), we should let the AI look up the instructions while it works.

They created a system called R-WoM (Retrieval-Augmented World Model).

Think of it this way:

• Old Way (Pure AI): The assistant tries to remember how to use Microsoft Word from memory. They guess, "I think I need to click the blue 'Insert' button." Click. Nothing happens. They guess again. Click. They get stuck.
• New Way (R-WoM): The assistant sees the task. Before guessing, they quickly pull up a digital tutorial (like a WikiHow article or a software manual) on a second screen. They read the exact steps: "To insert an image, go to the 'Insert' tab, then 'Pictures'." Then, they simulate the future while reading the manual.

How It Works (The 3-Step Magic)

The paper breaks this down into three clever tricks:

1. The "Smart Search" (Retrieval)

When the AI gets a task, it doesn't just guess. It acts like a librarian.

• Query Rewriting: If you ask, "How do I fork ChatGPT?", the AI rewrites that into a clearer search query like, "How to create a copy of a Git repository." This helps it find the right manual.
• Reranking: It finds 10 manuals but uses a smart filter to pick the best one, throwing away the ones that are about "forking a tree" instead of "forking code."

2. The "Long Daydream" (Simulation)

Once the AI has the right manual, it runs a simulation.

• Instead of just guessing one step, it uses a "Long Chain of Thought" (a fancy way of saying it thinks through the whole process in one go).
• It imagines: "If I click here, the menu opens. Then I click there, the file browser appears."
• Crucially, it checks every imagined step against the manual it just read. If the manual says "Click 'Open'" but the AI imagines "Click 'Cancel'," the manual corrects the AI.

3. The "Tournament" (Reward Estimation)

Usually, AI tries to give a score (like 8/10) to a plan. But that's hard to get right.

• R-WoM's Trick: Instead of scoring one plan, it generates three different plans and asks the AI: "Which of these three looks like it will actually work?"
• It's like a sports tournament. You don't need to know the exact score of every game; you just need to know which team is the best. This makes the AI much more stable and less likely to make mistakes.

Why This Matters (The Results)

The researchers tested this on two big challenges:

1. WebArena: Navigating complex websites (like buying things or managing forums).
2. OSWorld: Using desktop software (like Photoshop, Excel, or Linux terminals).

The Results:

• The new system (R-WoM) was significantly better than the old systems.
• On some tasks, it improved success rates by 23%.
• Most importantly, it got much better at long tasks. The old AI would get lost after 2 or 3 steps. The new AI, with its "manual" in hand, could successfully plan 3 or 4 steps ahead without getting confused.

The "Tutorial-Scarce" Bonus

What if there is no manual for a specific new software?
The paper also showed that the AI can write its own manuals. If the AI successfully completes a task once, it can write a tutorial for itself. Next time, it can read its own "self-written manual" to do the task again. This is like a student taking notes after a test and studying those notes for the next exam.

Summary

R-WoM is like giving a super-smart AI a GPS and a User Manual while it drives a car.

• Without it, the AI is a driver who relies on memory and often crashes because the road changed.
• With R-WoM, the AI checks the map (retrieval), follows the turn-by-turn directions (simulation), and picks the best route (ranking).

This makes AI agents much more reliable for doing real-world computer tasks, from organizing files to navigating the web, without getting stuck in their own daydreams.

Technical Summary

Here is a detailed technical summary of the paper "R-WOM: RETRIEVAL-AUGMENTED WORLD MODEL FOR COMPUTER-USE AGENTS" (ICLR 2026).

1. Problem Statement

Large Language Models (LLMs) are increasingly used as "world models" for computer-use agents to simulate future states and predict action outcomes, thereby reducing costly trial-and-error exploration. However, the authors identify a fundamental limitation: LLMs suffer from hallucinations and rely on static, parametric knowledge, which leads to compounding errors during long-horizon planning.

In complex digital environments (e.g., operating systems, web browsers), agents often generate procedurally coherent but ultimately infeasible steps because they lack environment-specific, up-to-date knowledge. The paper posits that while LLMs can predict immediate next states, they fail to maintain accuracy over extended planning horizons without external grounding.

2. Preliminary Analysis & Motivation

Before proposing a solution, the authors conducted a systematic probing of state-of-the-art LLMs (Qwen-2.5-VL-72B, Claude-3.5/3.7-Sonnet) on three core world model capabilities:

1. Next-State Identification: Predicting the immediate observation after an action.
   • Result: High accuracy (>75%). LLMs capture short-term dynamics well.
2. Full-Procedure Planning Alignment: Generating multi-step plans consistent with real-world tutorials.
   • Result: Moderate to low accuracy (<65% without retrieval). Performance degrades rapidly as the planning horizon increases, revealing an inability to align with specific environment constraints.
3. Milestone Transition Recognition: Evaluating whether a sequence of transitions leads to task progress (reward estimation).
   • Result: High accuracy (>83%). LLMs can recognize promising vs. unproductive paths when the context is clear.

Key Insight: LLMs possess strong short-term predictive and local evaluative skills but lack the specific procedural knowledge required for reliable long-horizon simulation in dynamic environments.

3. Methodology: R-WoM Framework

The authors propose Retrieval-augmented World Model (R-WoM), a framework that grounds LLM simulations in external, factual knowledge (tutorials) to mitigate hallucinations.

Core Components:

• Reasoning-Aware Retrieval Pipeline (RAG):
  • Instead of simple embedding similarity, R-WoM uses a two-stage retrieval process:
    1. Query Rewriting: An LLM rewrites the task query to be more comprehensive and general (e.g., anonymizing specific file names) to improve retrieval relevance.
    2. LLM-based Reranking: A list-wise reranker scores retrieved tutorial chunks based on semantic relevance to the specific task context, filtering out noise.
• Simulation with Long Chain-of-Thought (LongCoT):
  • Unlike prior works that use iterative rollouts (multiple model calls), R-WoM uses a single forward pass with LongCoT reasoning to simulate a $k$-step future trajectory. This improves efficiency while maintaining reasoning depth.
  • The world model conditions its simulation on the retrieved tutorial evidence ($E$) to predict state transitions.
• Adaptive Action Branching & Deduplication:
  • Branching: The policy model only generates multiple candidate actions when uncertain; otherwise, it proposes a single high-confidence action to save compute.
  • Deduplication: A verifier prunes semantically redundant candidates before simulation.
• Listwise Reward Estimation:
  • Instead of assigning absolute scores (e.g., 0, 0.5, 1) to trajectories, R-WoM uses a listwise ranking mechanism. The world model ranks all candidate rollouts relative to each other. This reduces bias from sparse absolute rewards and stabilizes action selection.

Workflow:

1. Given a task and current observation, retrieve and rerank relevant tutorials.
2. Generate candidate actions.
3. For each candidate, simulate a $k$-step future trajectory using the world model grounded by the retrieved tutorials.
4. Rank the simulated trajectories using listwise reward estimation.
5. Execute the highest-ranked action.

4. Key Contributions

1. Systematic Probing of LLMs as World Models: The paper provides empirical evidence that LLMs fail at long-horizon procedural planning without grounding, despite strong short-term prediction capabilities.
2. R-WoM Framework: A novel architecture integrating retrieval-augmented generation with world modeling, featuring LongCoT simulation and listwise reward estimation.
3. Tutorial Synthesis for Scarce Domains: A method to synthesize tutorials from self-played trajectories (using AgentNet data) for environments where official documentation is unavailable, extending R-WoM's applicability.
4. Efficiency Optimization: Introduction of adaptive branching and deduplication strategies that significantly reduce computational costs (token usage) without sacrificing performance.

5. Experimental Results

The framework was evaluated on two challenging benchmarks: WebArena (web tasks) and OSWorld (desktop OS tasks).

• Performance Gains: R-WoM consistently outperformed strong baselines (Vanilla LLM, RAG-only, and WebDreamer).
  • OSWorld: Achieved up to 23.4% relative improvement over the best baseline (Claude-3.7-Sonnet).
  • WebArena: Achieved up to 16.3% relative improvement.
• Imagination Horizon: R-WoM maintained high success rates as the imagination horizon increased (up to 3 steps), whereas ungrounded models (WebDreamer) saw performance plateau or decline due to error compounding.
• Ablation Studies:
  • Grounding Quality: Performance improved monotonically from no grounding $\to$ retrieved tutorials $\to$ oracle (human-annotated) tutorials.
  • Reward Design: Listwise reward estimation significantly outperformed absolute reward variants.
  • Cost-Efficiency: The adaptive version of R-WoM reduced token usage by >50% compared to full R-WoM while retaining most performance gains.
• Scarcity Adaptation: In tutorial-scarce scenarios, R-WoM using synthesized tutorials still outperformed baselines, demonstrating robustness.

6. Significance and Conclusion

This paper addresses a critical bottleneck in autonomous computer-use agents: the inability of LLMs to reliably simulate long-term consequences in dynamic, specific environments. By grounding world models in external, up-to-date tutorials, R-WoM bridges the gap between general LLM reasoning and specific procedural execution.

The work demonstrates that retrieval-augmented grounding is essential for scaling agent capabilities in complex digital environments. It offers a practical, efficient, and robust framework that reduces hallucinations and enables agents to perform multi-step tasks with higher success rates, paving the way for more reliable autonomous computer users.