💬 NLP

Agent Learning via Early Experience

This paper introduces "early experience," a paradigm where language agents learn from their own interaction data without explicit reward signals through implicit world modeling and self-reflection, demonstrating improved effectiveness and generalization while bridging the gap between supervised fine-tuning and reinforcement learning.

Original authors: Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, X

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kai Zhang, Xiangchao Chen, Bo Liu, Tianci Xue, Zeyi Liao, Zhihan Liu, Xiyao Wang, Yuting Ning, Zhaorun Chen, Xiaohan Fu, Jian Xie, Yuxuan Sun, Boyu Gou, Qi Qi, Zihang Meng, Jianwei Yang, Ning Zhang, Xian Li, Ashish Shah, Dat Huynh, Hengduo Li, Zi Yang, Sara Cao, Lawrence Jang, Shuyan Zhou, Jiacheng Zhu, Huan Sun, Jason Weston, Yu Su, Yifan Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to navigate a giant, complex maze. In the past, there were two main ways to teach this robot:

The "Shadowing" Method (Imitation Learning): You hire a human expert who has already solved the maze perfectly. You make the robot watch the expert walk the path over and over again. The robot memorizes the steps: "Turn left at the red door, then go straight."
- The Problem: The robot only knows the one path the expert took. If the expert made a tiny mistake, or if the robot gets slightly off-track, it panics because it has no idea what happens if it turns right instead of left. It's like a student who memorized the answers to a practice test but fails the real exam because the questions are slightly different.
The "Trial and Error" Method (Reinforcement Learning): You let the robot run the maze on its own. Every time it hits a wall, it gets a "zap" (a negative reward). Every time it finds the exit, it gets a "cookie" (a positive reward).
- The Problem: In the real world (like browsing a website or using a tool), there are no "zaps" or "cookies" waiting at every step. You might click a button and nothing happens, or you might fill out a form and not know if you did it right until days later. Without clear feedback, the robot spins its wheels, trying millions of random things, which is incredibly slow and inefficient.

The New Idea: "Early Experience"

This paper proposes a clever middle ground called Early Experience.

Think of it like this: Instead of just watching the expert, the robot is allowed to take a few "practice swings" on its own before it starts the real training.

Here is how it works, using two specific strategies the authors developed:

1. The "What-If" Simulator (Implicit World Modeling)

Imagine the robot is standing at a fork in the road.

The Expert says: "Go left."
The Robot asks: "What happens if I go right?" or "What happens if I jump?"

In the Early Experience method, the robot actually tries these "what-if" moves in a safe simulation. It sees the result: "Oh, if I go right, I fall into a pit," or "If I jump, I hit a ceiling."

The robot doesn't need a "cookie" or a "zap" to learn this. It simply learns the rules of the world. It builds an internal map of cause-and-effect: Action A leads to Result X. By predicting what happens next, the robot understands the environment much better than a robot that just memorized the expert's path.

2. The "Self-Reflection" Journal (Self-Reflection)

Imagine the robot is practicing a speech.

The Expert says: "Say 'Hello'."
The Robot tries: "Say 'Goodbye'."
The Result: The audience looks confused.

In this strategy, the robot looks at its own "mistakes" and the expert's "correct" choice side-by-side. It then uses its own brain (a large language model) to write a little note to itself: "I said 'Goodbye', but the expert said 'Hello'. The audience looked confused when I said 'Goodbye', but they smiled when the expert said 'Hello'. Therefore, 'Hello' is better because it fits the situation."

The robot isn't just memorizing the word "Hello"; it is learning the reasoning behind why that choice was better. It turns its own failures into a lesson plan.

Why This Matters (The Results)

The researchers tested this idea on eight different "mazes," including:

Navigating websites (like shopping for a specific shirt).
Solving science experiments.
Planning complex travel itineraries.
Using computer tools.

The findings were clear:

Better than just watching: Robots trained with "Early Experience" were much better at solving problems than robots that just watched experts.
Better at handling surprises: When the robots faced new situations they hadn't seen before (out-of-domain), they didn't crash as often. They understood the rules of the game, not just the moves.
Less data needed: They achieved high performance using only a fraction of the expert data usually required.
A bridge to the future: Even when clear rewards are available later (like in video games), starting with "Early Experience" made the robots learn even faster.

The Big Picture

The paper argues that we don't need to wait for perfect "reward systems" (like cookies and zaps) to be invented for every real-world task. We can teach agents to learn from their own "what-if" scenarios and their own reflections on mistakes right now. It's a way to turn a robot from a passive student who memorizes answers into an active learner who understands how the world works.

Technical Summary: Agent Learning via Early Experience

1. Problem Statement

The long-term goal of language agents is to learn and improve through their own experience in complex, real-world tasks. However, current training paradigms face significant limitations:

Reinforcement Learning (RL): While successful in domains with verifiable rewards (e.g., games), applying RL to real-world language agents is difficult. Many environments (e.g., websites, multi-turn tool use) lack verifiable or dense reward signals, or require inefficient long-horizon rollouts that make credit assignment unstable.
Supervised Fine-Tuning (SFT/Imitation Learning): Most current agents rely on SFT using expert-curated data. While efficient, this approach has inherent limitations:
- Distribution Shift: Agents do not interact with the environment during training, so they never observe the consequences of non-expert actions. This leads to compounding errors when the agent deviates from the training distribution.
- Limited Generalization: Expert demonstrations capture only a narrow range of scenarios, restricting the agent's ability to generalize to unseen situations or recover from failures.
- Scalability: High-quality expert demonstrations are costly to scale.

The core challenge is how to train agents to grow from their own experience without relying on external reward signals or being strictly bound by the coverage of human demonstrations.

2. Methodology: The Early Experience Paradigm

The authors propose Early Experience, a middle-ground paradigm between imitation learning and reinforcement learning. In this setting, agents learn not only from human-curated data but also from future states driven by their own proposed actions in the environment. These future states serve as supervision signals without requiring external rewards.

The paradigm leverages a dataset of expert trajectories $D_{expert} = \{(s_i, a_i)\}$ and augments it with a rollout dataset $D_{rollout} = \{(s_i, a^j_i, s^j_i)\}$ , where $a^j_i$ are alternative actions sampled from the agent's policy, and $s^j_i$ are the resulting next states observed in the environment.

Two specific training strategies are studied under this paradigm:

A. Implicit World Modeling (IWM)

Concept: The agent treats the collection of observed future states as an auxiliary prediction task to internalize environment dynamics.
Mechanism: For each state-action pair $(s_i, a^j_i)$ in the rollout dataset, the model is trained to predict the resulting next state $s^j_i$ .
Objective: The training objective is a next-token prediction loss: $L_{IWM} = -\sum \log p_\theta(s^j_i | s_i, a^j_i)$ .
Benefit: This allows the policy to learn regularities in environment behavior (transitions, side effects, invalid outcomes) directly, grounding the agent in its operating context without a separate simulator module.

B. Self-Reflection (SR)

Concept: The agent learns from its suboptimal actions by comparing them to expert actions and generating natural language explanations for why the expert choice is superior.
Mechanism: For each expert state $s_i$ , the agent executes the expert action $a_i$ to get $s_{i+1}$ and alternative actions $a^j_i$ to get $s^j_i$ . The model is then prompted to generate a chain-of-thought explanation $c^j_i$ justifying why $a_i$ is preferable to $a^j_i$ based on the differences in outcomes.
Objective: The model is trained to jointly predict the reasoning $c^j_i$ and the expert action $a_i$ given the state $s_i$ : $L_{SR} = -\sum \log p_\theta(c^j_i, a_i | s_i)$ .
Benefit: This extracts fine-grained, transferable decision principles (e.g., constraint satisfaction) from the agent's own exploratory failures, moving beyond rote imitation.

3. Key Contributions

Formalization of Early Experience: The paper advocates for and formalizes early experience as a scalable, reward-free bridge between imitation learning and RL. It empowers agents to convert their own interactions into learning signals without external rewards.
Two Training Strategies: The authors propose and systematically study Implicit World Modeling (internalizing dynamics via state prediction) and Self-Reflection (distilling decision principles via reasoning about suboptimal actions).
Comprehensive Evaluation: The methods are evaluated across eight diverse environments (embodied navigation, web navigation, multi-turn tool use, scientific simulation, and planning) and multiple model families (Llama-3.2-3B, Qwen-2.5-7B, Llama-3.1-8B).

4. Experimental Results

The evaluation demonstrates consistent improvements over pure imitation learning baselines:

Effectiveness: Across all eight benchmarks and model sizes, both IWM and SR outperform standard SFT.
- IWM shows steady gains in structured environments with predictable dynamics (e.g., ALFWorld, ScienceWorld, WebShop).
- SR delivers significant improvements in tasks requiring multi-step reasoning and constraint satisfaction (e.g., TravelPlanner, BFCLv3).
- Notably, on WebShop, IWM improved success rates by up to 18.4% over imitation learning.
Out-of-Domain (OOD) Generalization: Early experience methods consistently recover a substantial portion of the performance gap in OOD splits. In some cases (e.g., SearchQA), the relative gains in OOD settings exceed those in in-domain settings, indicating that learning from one's own rollouts prepares agents for states not covered by demonstrations.
Data Efficiency: Early experience maintains a lead over imitation learning even when using significantly fewer expert demonstrations (e.g., 1/8 of the data on WebShop).
Model Scaling: The benefits persist as model size scales up to 70B parameters (Llama-3.3-70B), with early experience checkpoints consistently occupying the top performance curve.
Foundation for RL: When subsequent RL (using GRPO) is applied to checkpoints trained with early experience, the final performance ceilings are substantially higher than those starting from pure imitation learning checkpoints. This suggests early experience provides a robust initialization for the "era of experience."

5. Significance and Claims

The paper positions Early Experience as a necessary stepping stone for the development of autonomous language agents. Its significance lies in:

Bridging the Gap: It offers a practical solution for environments where rewards are unavailable or unreliable, allowing agents to learn from consequences rather than just static demonstrations.
Scalability: By utilizing the agent's own actions to generate supervision, it reduces the dependency on costly, high-quality human demonstrations.
Future Readiness: It acts as a "mid-training" bridge that prepares agents for fully experience-driven learning (RL) once verifiable rewards become available in new environments, unlocking further gains without retraining from scratch.

The authors conclude that early experience enables agents to grow from their own interactions now, addressing the limitations of current imitation-based approaches while laying the groundwork for the future era of experience-driven agents.