💬 NLP

ProPlay: Procedural World Models for Self-Evolving LLM Agents

ProPlay introduces a procedural world model that enables self-evolving LLM agents to improve in partially observable environments by abstracting successful trajectories into a causal procedure graph for pre-episode simulation and post-episode refinement, thereby enhancing environment understanding and autonomous learning without external supervision.

Original authors: Yijun Ma, Zehong Wang, Yiyang Li, Ziming Li, Xiaoguang Guo, Weixiang Sun, Chuxu Zhang, Yanfang Ye

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yijun Ma, Zehong Wang, Yiyang Li, Ziming Li, Xiaoguang Guo, Weixiang Sun, Chuxu Zhang, Yanfang Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Mental Rehearsal" Agent

Imagine you are trying to learn how to cook a complex new dish in a kitchen you've never seen before. You can't see the whole room at once (it's "partially observable"), and you only get feedback when you taste the food or when something burns.

Most current AI agents are like chefs who either:

Memorize recipes (Memory): They remember every single step they did before, but they don't really understand why the steps work together.
Guess and check (Planning): They try to think of every possible move before making one, but they often get stuck because they don't have a good map of the kitchen.

ProPlay is a new kind of AI agent that tries to do what humans do: Mental Rehearsal (Preplay).

Before the agent even touches a tool, it closes its eyes (metaphorically) and imagines the whole process. It asks, "If I do X, then Y usually happens, and then I can do Z." It builds a mental map of how tasks flow, not just a list of rules.

How ProPlay Works: The Three-Step Loop

The paper describes a cycle that happens every time the agent tries a new task. Think of it like a student studying for a test, taking the test, and then reviewing their notes.

1. The "Mental Map" (The Procedural World Model)

Instead of remembering every single tiny action (like "pick up spoon," "move spoon 2 inches"), ProPlay groups actions into Procedures.

Analogy: Think of a procedure like a "chapter" in a book. Instead of remembering every word on page 10, you remember the chapter title: "The Hero Finds the Sword."
The Graph: ProPlay connects these chapters into a map (a graph). It draws arrows showing that "Finding the Sword" usually leads to "Fighting the Dragon."
The Reliability Score: Crucially, ProPlay puts a "trust score" on every arrow. If the "Hero finds the sword" step led to a win 9 times out of 10, that arrow gets a high score. If it led to a trap, the score drops. This helps the agent know which paths are safe to trust.

2. The "Preplay" (Rehearsal)

Before the agent starts the actual task, it looks at its map and runs a simulation in its head.

Soft Guidance: It doesn't force the agent to follow the map like a robot. Instead, it says, "Hey, based on what I've learned, this path usually works well. Try it, but if you see something weird, feel free to change course."
Why this matters: It gives the agent a head start (exploitation) but still lets it explore new things if the map is wrong (exploration).

3. The "Review" (Refinement)

After the agent finishes the task, it looks at what actually happened.

Updating the Map: If the agent succeeded, ProPlay strengthens the "trust scores" on the path it took. If it failed, it marks that path as risky.
Learning New Chapters: If the agent did something new that worked, ProPlay creates a brand new "chapter" (procedure) and adds it to the map for next time.

Why Is This Better? (The Results)

The researchers tested ProPlay on three different "games" (benchmarks):

ScienceWorld: A text-based game where you have to solve science problems.
τ-Bench: A simulation of helping a customer with a retail or airline issue.
PlanCraft: A Minecraft-like game where you have to craft items using recipes.

The Findings:

Better at Complex Tasks: ProPlay beat other top AI agents, especially in tasks that required long chains of steps (like mixing chemicals or crafting complex items).
Faster Learning: In the beginning (the "cold start"), ProPlay learned faster because it could use its mental map to guess good strategies, even without much experience.
The "Sweet Spot": The paper found that ProPlay is great at tasks with clear steps (like a recipe), but it struggles with tasks that require precise math or building tools from scratch that don't fit a pattern.

The Limitations (What the Paper Says)

The authors are honest about where ProPlay might stumble:

Too Abstract: If a task needs very specific numbers (like "measure exactly 3.4 grams"), ProPlay's high-level "chapters" might be too vague. It's better at "add salt" than "add 3.4g of salt."
One-Time Rehearsal: The agent only does the "mental rehearsal" once at the very start. If the situation changes halfway through, it can't stop and re-rehearse; it has to rely on its own wits to fix the plan.
New Environments: The paper hasn't tested this on open-ended things like browsing the whole internet, where the rules change constantly and might not form a clear "map."

Summary

ProPlay is an AI that learns by building a mental map of "how things usually happen." It rehearses these paths before acting, trusts the paths that have worked before, and updates its map after every attempt. It's like a chef who doesn't just memorize a recipe but understands the flow of cooking, allowing them to adapt when the kitchen is messy or the ingredients are slightly different.

Technical Summary: ProPlay: Procedural World Models for Self-Evolving LLM Agents

Problem Statement

The paper addresses the challenge of building self-evolving LLM agents that can improve through interaction without external supervision or large-scale retraining. This is particularly difficult in partially observable environments, where agents must actively explore, learn from sparse and delayed feedback, and decide when to trust prior experience.

Existing approaches typically treat memory (storing reflections or rules) and planning (searching future actions) as separate modules, failing to close the loop between them to continually refine an internal understanding of environment dynamics. Furthermore, current world model methods often rely on action-level constraints or low-level rules, which may miss the higher-level causal structure of tasks and struggle to balance exploitation with exploration. The core problem is how to represent environment dynamics at a level of abstraction that captures reusable, hierarchical task progressions while allowing for flexible low-level action choices.

Methodology: ProPlay

The authors introduce ProPlay, a framework built on an evolving procedural world model. The system operates in an episodic three-phase loop: Preplay, Execution, and Refinement.

1. Procedural World Model

Instead of representing the environment via individual actions or entities, ProPlay models dynamics using procedures as the fundamental unit.

Procedures: High-level abstractions of task stages (e.g., "boiling water," "measuring a property") induced from successful interaction trajectories. They describe the purpose and execution template of a stage.
Procedure Graph ( $G$ ): An attributed directed graph where nodes are procedures ( $P$ ) and directed edges ( $E$ ) encode causal transitions between task stages.
Reliability Record Embeddings ( $C$ ): Each transition $(p_i, p_j)$ is associated with an embedding that quantifies how consistently that transition contributed to successful tasks with similar descriptions. This is calculated as a weighted summation of task description embeddings from past successful episodes where the transition occurred. This mechanism helps distinguish generalizable knowledge from noisy or overfitted experience.

2. Procedure-Level Preplay

Before each new episode, ProPlay performs procedure-level preplay to generate structured soft guidance:

Trajectory Construction: The agent queries the current procedure graph ( $G_{k-1}$ ) and retrieves relevant failure experiences. It assigns task-specific reliability scores to existing transitions based on semantic similarity to the current task description.
Soft Guidance: The agent constructs a procedure trajectory ( $W^*$ ) by reasoning over the task description, the graph, and reliability scores. This trajectory is injected into the agent's context as soft guidance (a high-level prior), not a hard constraint. This allows the agent to exploit accumulated knowledge while retaining the freedom to deviate and explore unseen transitions if observations warrant it.

3. Episodic World Model Refinement

After the episode concludes, ProPlay refines the world model based on environment feedback:

Procedure Induction: Successful trajectory segments are partitioned and passed to the LLM to induce new procedures or merge with existing ones based on semantic similarity, ensuring a non-redundant procedural vocabulary.
Graph Update: New procedure pathways are mapped to the updated set of procedures, and new transitions are added to the graph.
Reliability Update: The reliability record embeddings for observed transitions are updated to reflect their contribution to the current episode's reward, consolidating generalizable knowledge and suppressing misleading transitions.

Key Contributions

Procedure-Centric Perspective: The paper proposes modeling environment dynamics as procedural pathways rather than low-level action constraints, capturing the hierarchical and causal structure of task progressions.
ProPlay Framework: A preplay mechanism that constructs and maintains a procedural world model, retrieving task-relevant trajectories as soft guidance to balance exploration and exploitation.
Empirical Evaluation: Comprehensive experiments on three public benchmarks demonstrating consistent improvements in environment understanding and self-evolution capabilities.

Experimental Results

The authors evaluated ProPlay on ScienceWorld (scientific reasoning), $\tau$ -Bench (conversational tool-use), and PlanCraft (multi-step crafting).

Performance: ProPlay achieved the best Average Ranking (A.R.) across all benchmarks, outperforming strong baselines including ReAct, Reflexion, ExpeL, LATS, Wall-E, and WorldCoder.
ScienceWorld: ProPlay significantly outperformed planning-based and action-level world model baselines, particularly in complex multi-step tasks.
Self-Evolution Analysis:
- Cold-Start: ProPlay showed advantages in early episodes due to its ability to leverage prior knowledge for exploration.
- Growth Pattern: The procedural graph exhibited a two-phase growth: rapid accumulation of concept nodes early on, followed by a proliferation of relational edges, suggesting that deepening relational structure drives late-stage performance.
- Generalization: The system generalized well across heterogeneous tasks, showing significant advantages in tasks where it excelled without falling behind in others.
Ablation Studies:
- Preplay: Replacing soft guidance with hard constraints or random sampling reduced performance, confirming the necessity of flexible, grounded reasoning.
- World Model Components: Removing procedure transitions or reliability records significantly degraded performance, highlighting that transitions are the most valuable knowledge and reliability records are crucial for effective exploitation.
- Granularity: Replacing the procedural model with an action-level model resulted in lower performance, indicating that action-level constraints limit reasoning flexibility.

Limitations and Discussion

The paper acknowledges specific limitations:

Scope of Abstraction: Procedural abstraction is most effective for tasks with clear, reusable causal structures. Tasks requiring precise quantitative reasoning (e.g., melting point measurement) or non-decomposable tool assembly may be better served by fine-grained rule-based methods.
Single-Shot Preplay: The preplay plan is generated once at the start of an episode. If early observations reveal the plan is incorrect, the agent cannot revise the plan mid-episode and must rely on its own reasoning to recover.
Benchmark Scope: It remains an open question whether these benefits generalize to highly open-ended environments (e.g., web navigation, GUI interaction) where procedural structures may not emerge naturally.

Significance

The paper claims that procedure-level abstraction offers a principled and effective foundation for self-evolving agents in partially observable environments. By closing the loop between memory and planning through a procedural world model, ProPlay enables agents to internalize environment dynamics, distinguishing generalizable knowledge from contingent experience. This approach moves beyond isolated rule extraction or static planning, providing a dynamic framework where agents can actively rehearse future paths and refine their internal understanding of the world through interaction.