Technical Summary: Bridging the Agent-World Gap: Text World Models for LLM-based Agents

1. Problem Statement

Large Language Model (LLM)-based agents are increasingly deployed in interactive textual environments, including web navigation, code editing, tool use, and long-horizon dialogue. However, a critical limitation persists: many agents operate reactively, mapping observations to actions without an explicit, inspectable model of how these environments are structured and evolve. This "agent-world gap" leads to failures in planning, inefficient learning, and a lack of principled evaluation. For instance, a web agent may click links without modeling page evolution, or a coding agent may patch files without simulating runtime propagation.

To address this, the paper introduces Text World Models (TWMs). Unlike traditional world models that predict pixels or state vectors, TWMs are transition models over textual states. Given a current textual state (e.g., a dialogue history, terminal trace, or webpage) and a candidate action, a TWM predicts the resulting textual state (e.g., a user reply, API response, or terminal output). The paper argues that the textual setting introduces unique tensions absent in visual domains: states are open-vocabulary and ambiguous, dynamics rely heavily on background knowledge, and correctness is often subjective and semantic rather than binary.

2. Methodology and Framework

The paper provides a systematic review of the field, organized around a formal framework and the agent lifecycle.

2.1 Formal Definition and Taxonomy

A text world model is defined as a transition function $M: S \times A \rightarrow T_S$, where $S$ is the agent-visible state, $A$ is the action, and $T_S$ is the textual rendering of the successor state. The authors propose a two-axis taxonomy to characterize existing works:

1. State/Transition Representation:
   • Natural Language: Free-form descriptions (flexible but weak on consistency).
   • Structured: Explicit schemas like JSON, knowledge graphs, or PDDL (easier to track/verify).
   • Executable Code: Python, TypeScript, or HTML programs (deterministic, verifiable, reproducible).
2. Grounding Domain:
   • Physical: Embodied environments, game physics, disaster scenes.
   • Digital: Web interfaces, code repositories, APIs, terminals.
   • Social: Human preferences, dialogue, and intent.
   • Abstract: Planning, math, and symbolic reasoning.

2.2 Construction Paradigms (§3)

The paper categorizes how TWMs are built into three paradigms:

• LLM-as-World-Model (Learning-Based): The LLM itself is the transition function.
  • Supervised Fine-Tuning (SFT): Trained on $(s, a, s')$ trajectories. Approaches vary between full-state prediction (generating the entire next observation) and delta prediction (generating only the change, which is more efficient for large observation spaces like web pages). Data sources range from real environment rollouts to synthetic self-play or repurposed human conversations.
  • Reinforcement Learning (RL): Optimizes for prediction quality rather than token likelihood. Rewards range from surface fidelity (string matching) to semantic equivalence (embedding similarity) and behavioral consistency (ensuring the predicted state leads to the same agent decision).
• LLM-as-World-Model (Prompt-Based): Uses frozen LLMs with no parameter updates.
  • In-Context: Relies on the model's prior knowledge (limited by the model's training data).
  • Retrieval-Augmented: Grounds predictions in external procedural traces or structured knowledge bases.
  • Self-Evolving: The agent accumulates knowledge through interaction, distilling rules or memories into the prompt context.
• Code-as-World-Model: The LLM acts as a coder, generating executable code (PDDL, Python, HTML) that serves as the world model. This paradigm offers deterministic transitions and formal verification, separating the LLM's role (writing code) from the world model's role (executing it).

2.3 Application Lifecycle

The survey examines how TWMs are utilized at different stages of the agent lifecycle:

• Training-Time (§4):
  • Internalization: Folding the world model into the agent's parameters (e.g., as a warm-start or within the reasoning trace) to enable anticipation without external simulators.
  • Training Environments: Using TWMs as synthetic environments for offline data synthesis or online RL, reducing reliance on costly real-world access.
  • User Simulation: Simulating human users to train agents in interactive, multi-turn scenarios where real users are too sparse or stochastic.
• Inference-Time (§5):
  • Simulator (Lookahead): Using the TWM to simulate candidate futures. This includes shallow lookahead (scoring immediate consequences) and deep tree search (using TWMs as transition functions within MCTS or similar algorithms).
  • Verifier: Using the TWM to screen, rank, or rewrite actions proposed by the policy before execution, mitigating hallucination and safety risks.

2.4 Evaluation (§6)

The paper outlines evaluation methodologies from two perspectives:

1. Evaluating the World Model: Measuring prediction accuracy (single-step exact match, multi-step consistency) and task-driven utility (does the model improve downstream agent performance?). The authors note that high prediction accuracy does not always correlate with task success.
2. World Model as Evaluation Tool: Using TWMs to benchmark agents in simulated environments or with simulated users. A critical finding is the simulator validity problem: simulated users often overestimate agent competence due to excessive cooperativeness, necessitating dedicated user models trained on real human data.

3. Key Contributions

1. First Systematic Review: This is the first survey to treat text world models for LLM agents as a distinct object of study, moving beyond visual/control-centric surveys.
2. Unified Framework: It establishes a formal definition of TWMs and a two-axis taxonomy (representation vs. grounding domain) to organize fragmented literature across web, code, and dialogue domains.
3. Lifecycle Perspective: It structures the field around the agent lifecycle (Construction $\rightarrow$ Training $\rightarrow$ Inference $\rightarrow$ Evaluation), clarifying how design choices in one stage impact others.
4. Paradigm Comparison: It provides a detailed cross-paradigm comparison of Learning-based, Prompt-based, and Code-based approaches, highlighting their trade-offs in fidelity, verifiability, and adaptability.
5. Identification of Open Challenges: The paper synthesizes limitations such as the "sim-to-real" gap, the lack of transferable verification standards for synthesized environments, and the need for reasoning-capable world models.

4. Results and Findings

• Construction: Learning-based methods (SFT/RL) achieve high fidelity in structured environments but struggle with long-horizon consistency. Prompt-based methods offer rapid adaptation but suffer from hallucination. Code-based methods provide verifiable, deterministic transitions but require domain-specific code generation.
• Training: Internalizing world models improves deployment efficiency but limits simulation at inference time. Synthetic environments (offline/online) can match or exceed real-environment training performance, but "hallucination drift" remains a risk when agents explore out-of-distribution states.
• Inference: Tree search methods using TWMs (e.g., LLM-MCTS) can outperform larger models reasoning without search. Verifiers effectively filter catastrophic actions but rely heavily on the simulator's fidelity.
• Evaluation: Single-step prediction accuracy often saturates in structured environments but collapses in open-ended ones. Crucially, task success does not guarantee world understanding; agents can complete tasks via pattern matching without modeling the environment correctly. Furthermore, current benchmarks often overestimate agent capability due to overly cooperative simulated users.

5. Significance and Claims

The paper claims to consolidate a rapidly developing area, clarifying its design space and providing a common vocabulary for researchers. Its primary significance lies in shifting the focus from reactive agents to anticipatory agents equipped with explicit models of their environment.

The authors modestly claim that this survey serves as a resource to "consolidate this rapidly developing area, clarify its design space, and highlight open challenges." They do not propose a new algorithm but rather a structured lens through which to view existing work. The paper emphasizes that while TWMs are essential for planning and efficient learning, the field faces significant hurdles in grounding (connecting text to physical reality), continual learning (adapting to non-stationary environments), and evaluation validity (ensuring simulators faithfully represent reality). The ultimate goal is to bridge the gap between LLM fluency and robust, structured environmental understanding.