💬 NLP

Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective

This paper identifies "exploration collapse" as a key failure mode where LLM agents lose the ability to discover new solutions in unfamiliar environments during reinforcement learning, and proposes SPA, a method that improves performance by first grounding the agent in state and transition prediction via self-experience supervised finetuning before optimizing for reward.

Original authors: Shiqi Chen, Tongyao Zhu, Zian Wang, Jinghan Zhang, Kangrui Wang, Ruochen Zhou, Siyang Gao, Teng Xiao, Yee Whye Teh, Junxian He, Manling Li

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiqi Chen, Tongyao Zhu, Zian Wang, Jinghan Zhang, Kangrui Wang, Ruochen Zhou, Siyang Gao, Teng Xiao, Yee Whye Teh, Junxian He, Manling Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer program designed to think and act like a human, capable of solving puzzles, navigating virtual worlds, or even using a web browser to find information. These programs, known as large language models, are trained on vast amounts of text from the internet, learning to predict what word comes next in a sentence. When researchers ask these models to act as agents—taking steps to achieve a goal—they often use a method called reinforcement learning. This is a process where the model tries different actions, receives feedback on whether it succeeded or failed, and gradually learns to repeat the actions that lead to success. It is a powerful way to teach a machine how to do things it was not explicitly programmed to do. However, a critical problem arises when these agents are placed in new, unfamiliar situations. While they might learn to solve a specific puzzle they have seen before, they often struggle to figure out how to explore a completely new environment, getting stuck in a narrow way of thinking that prevents them from finding the best solution.

A team of researchers set out to understand why these intelligent agents fail when they encounter the unknown. They discovered a specific phenomenon they call "exploration collapse." In simple terms, when an agent is trained on a new task, it often learns to find just one specific path to success and then ignores all other possibilities. It becomes so focused on that single route that it loses the ability to try different approaches. The researchers measured this by looking at two different scores: one that tracks if the agent's very best guess works, and another that tracks if any of its many different guesses work. In familiar tasks, both scores improve as the agent learns. But in new, difficult environments like a grid-based puzzle where boxes must be pushed to specific targets, the score for the single best guess goes up slightly, while the score for trying many different paths actually goes down. The agent is learning to be more confident in a bad habit rather than learning to be more flexible. This happens because the agent does not truly understand the rules of the new world it is in; it is guessing without a solid foundation.

To fix this, the researchers developed a new training method called SPA, which stands for Self-Experience Agent. Instead of immediately trying to teach the agent how to get rewards, they first taught it how to understand the world around it. They gave the agent a chance to explore the environment on its own, without the pressure of scoring points. During this phase, the agent was asked to describe what it saw and to predict what would happen next if it took a certain action. For example, if the agent saw a box in a specific spot and decided to push it, it had to first state where the box was and then predict where it would end up. The researchers then used the actual, correct outcomes from the environment to correct the agent's predictions, effectively teaching it the laws of physics for that specific game. This process created a kind of internal map or "world model" inside the agent's mind, grounding its understanding in reality rather than in vague guesses.

Once the agent had built this internal understanding of how the world works, the researchers then started the standard training process to teach it how to win. The results were striking. In tests on a puzzle game called Sokoban, where the agent had to push boxes to targets, the standard training method only allowed the agent to succeed about 25 percent of the time. With the new method, the success rate jumped to nearly 60 percent. On another puzzle involving a frozen lake, the success rate improved from about 22 percent to over 70 percent. Perhaps most surprisingly, a very small computer model trained with this method was able to outperform a much larger, more powerful model that had been trained using the old, standard methods. The small model, having learned the rules of the game first, was able to navigate the complex puzzles more effectively than the giant model that had just been trying to guess its way to a reward.

The researchers also tested whether this approach worked on other types of tasks, such as a logic puzzle called Sudoku and a simulated environment for household chores. In every case, the method that taught the agent to understand the state of the world before trying to win led to better results. They found that the key to success was not just having more data or a bigger computer, but rather the order in which the learning happened. By forcing the agent to learn the structure of the environment first, it avoided the trap of collapsing into a single, brittle strategy. The agent learned to keep its options open, exploring multiple paths because it understood the consequences of its actions. This approach suggests that for artificial intelligence to truly adapt to new and complex situations, it needs to build a reliable understanding of reality before it tries to optimize for success. The study shows that when an agent is grounded in the actual mechanics of its environment, it can learn to explore more effectively and solve problems that were previously out of reach.

Technical Summary: Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective

1. Problem Statement: Exploration Collapse in Unseen Environments

The paper identifies a critical failure mode in Agentic Reinforcement Learning (RL) for Large Language Models (LLMs) when deployed in environments unfamiliar to the model's pre-training distribution. While standard RL improves single-trajectory success rates (Pass@1) in familiar settings (e.g., ALFWorld, WebShop), it induces a phenomenon the authors term exploration collapse in unseen environments (e.g., Sokoban, FrozenLake, Sudoku).

In these unfamiliar domains, characterized by high state perplexity relative to the model's prior knowledge, standard RL causes Pass@k (the probability that at least one of $k$ sampled trajectories succeeds) to drop markedly as training progresses, even as Pass@1 edges up. This divergence indicates that the policy becomes increasingly brittle, converging on narrow, exploitative solution paths while losing the ability to diversify attempts or generalize robustly. The authors attribute this collapse to a lack of grounding in environment states and dynamics; the agent fails to internalize a "world model" of the underlying rules, leading to poor state parsing and ineffective credit assignment.

2. Methodology: SPA (Self-Experience Agent)

To mitigate exploration collapse, the authors propose SPA, an "explore-then-exploit" framework that grounds the agent before optimizing for rewards. Unlike classical model-based RL that queries an explicit world model during planning, SPA internalizes world modeling into the policy itself as a cold-start initialization.

The framework consists of two primary stages:

A. State Estimation (Grounding)

To address the distribution shift where raw symbolic state strings (e.g., grid maps) have high perplexity, SPA augments the raw state $s'_t$ with an abstract, structured representation $b_t$ .

Mechanism: The model is prompted to generate a structured natural language description of the current state (e.g., coordinates of the player, boxes, and goals) alongside the raw observation.
Goal: This reduces state perplexity and transforms unfamiliar symbolic inputs into familiar, interpretable representations, facilitating better reasoning.

B. Self-Experience Supervised Fine-Tuning (SFT)

Before RL, the agent undergoes a supervised fine-tuning stage using its own interaction trajectories to learn transition dynamics.

Data Collection: The base model freely explores the environment to collect trajectories $\tau = (s_0, o_0, a_0, \dots)$ .
World Modeling Objective: During reasoning (between <thought> and </thought> tags), the model is prompted to explicitly describe the current state (<observation>) and predict the next state (<prediction>).
Ground Truth Supervision: Crucially, the model's self-generated beliefs about the current and next states are replaced with ground-truth states ( $s_t, s_{t+1}$ ) provided by the environment simulator. The model is then trained via SFT to predict these grounded states and the subsequent action.
Loss Function: The training minimizes a masked token-level cross-entropy loss over the reasoning spans containing state descriptions and predictions, effectively teaching the agent the environment's transition kernel $p(s_{t+1} | s_t, a_t)$ .

C. RL Training

The SFT-trained model, now possessing a grounded internal world model, serves as the initialization for standard Proximal Policy Optimization (PPO). The RL phase optimizes for task rewards using the grounded state representations, without adding auxiliary world-modeling rewards that might interfere with the primary objective.

3. Key Contributions

Diagnosis of Exploration Collapse: The paper empirically characterizes the divergence between Pass@1 and Pass@k in unseen environments, identifying it as a grounding failure rather than a general RL inefficiency.
SPA Framework: It introduces a simple, scalable recipe that decouples world modeling (SFT) from reward optimization (RL), using self-experience data to initialize the policy.
Ablation and Mechanism Analysis: Through controlled studies, the authors isolate four critical factors driving SPA's success:
- Grounded State Representations: Abstracting raw states into structured coordinates.
- Explicit Transition Modeling: Learning to predict next states, not just actions.
- Self-Experience Trajectories: Collecting data from a sufficiently strong exploration policy (random actions fail to yield gains).
- Adequate Data Coverage: Ensuring sufficient transition data and format adherence.
Performance on Sub-3B Models: The method enables small models (e.g., Qwen2.5-1.5B) to significantly outperform much larger baselines (e.g., GPT-OSS-20B) on complex spatial reasoning tasks.

4. Experimental Results

The authors evaluated SPA across multiple model families (Qwen2.5, LLaMA-3.2, GPT-OSS) and environments (Sokoban, FrozenLake, Sudoku).

Performance Gains: On the Qwen2.5-1.5B-Instruct model, SPA increased the Sokoban success rate (Pass@1) from 25.6% (Vanilla RL) to 59.8%, and FrozenLake from 22.1% to 70.9%.
Restoration of Exploration: Unlike Vanilla RL, where Pass@k degrades, SPA consistently improves both Pass@1 and Pass@k (e.g., Sokoban Pass@8 rose from 34.0% to 69.5%).
Comparison with Baselines: SPA outperformed "State Estimation RL" (which lacks transition modeling) and VAGEN (a reward-shaping world-modeling baseline). The authors note that VAGEN's online reward shaping interferes with the RL objective, whereas SPA's offline SFT approach provides a cleaner initialization.
Generalization: SPA trained on simple variants (e.g., 4x4 FrozenLake) successfully transferred to harder variants (6x6), demonstrating "easy-to-hard" transfer. However, cross-game generalization (Sokoban to FrozenLake) remained difficult, suggesting the world model is specific to the environment family.

5. Significance and Claims

The paper claims that the primary barrier to scaling agentic RL in unfamiliar environments is not the lack of reward signals, but the lack of grounded state and transition understanding. By explicitly teaching the agent to model the environment's dynamics before optimizing for rewards, SPA restores the agent's ability to explore diverse solution paths.

The authors emphasize that SPA is a policy-centric approach where the world model is internalized as a prior, rather than an external simulator used at inference time. They conclude that grounding the policy via self-experience SFT is a necessary prerequisite for effective RL in out-of-distribution settings, offering a reproducible starting point for building robust agents in unfamiliar environments.

Limitations Acknowledged:
The authors note that their results are currently limited to grid-style interactive benchmarks and small-to-mid-scale open-weight models. They acknowledge that SPA relies on access to ground-truth states (privileged information) during the SFT stage, which may not be available in all real-world deployments without an additional perception module. Furthermore, cross-game generalization remains a challenge.