🤖 machine learning

Training Language Agents to Learn from Experience

This paper introduces the In-context Training (ICT) framework and a reinforcement learning-based pipeline that enables language agents to distill experience from past interactions into reusable system prompts, thereby achieving cross-task self-improvement and generalization to unseen environments without human supervision.

Original authors: Yuval Shalev, Zifeng Ding, Mateja Jamnik

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuval Shalev, Zifeng Ding, Mateja Jamnik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly naive, robot how to play video games.

The Problem: The "One-and-Done" Amnesia
Currently, these AI robots (called "Language Agents") are good at learning from their mistakes, but only in a very specific way. If they fail a level in a game, they can look back, say, "Oh, I hit the wrong wall," and try again immediately. They get better at that specific level.

However, once they beat that level, they forget everything. If you give them a new level that is slightly different, they act like they've never played the game before. They have to re-learn the lesson from scratch. It's like a student who studies hard for a math test, gets an A, and then walks into a physics class the next day as if they've never seen a number before. They can't "distill" the general lessons (like "always check your surroundings") into a reusable rulebook.

The Solution: The "Coach" and the "Player"
The authors of this paper introduced a new framework called In-context Training (ICT). Think of it as a two-person team:

The Player (The Actor): This is the robot trying to solve the tasks. It plays the game, makes mistakes, and collects a "replay" of what happened.
The Coach (The Reflector): This is a second AI that watches the Player's replays. The Coach's job isn't to play the game; it's to write a new instruction manual (a system prompt) for the Player.

Here is the magic: The Coach doesn't just say, "Don't hit that wall." It looks at a batch of different levels the Player just tried, finds common patterns, and writes a general rule for the Player to use on future, unseen levels.

The Training: Learning to Coach
The big question was: Can we teach the Coach how to write these better manuals?

The researchers didn't use human teachers or pre-written examples. Instead, they used a "trial and error" loop (Reinforcement Learning):

The Coach writes a manual.
The Player tries to solve a batch of new, unseen puzzles using that manual.
If the Player does well, the Coach gets a "good job" signal. If the Player fails, the Coach gets a "try again" signal.
The Coach learns from this feedback to write even better manuals next time.

The Results: From Chess to Cooking
The team tested this on two very different "video games" (simulated environments):

ALFWorld: A text-based house where the robot has to find objects, clean them, and put them in specific places (like "cool a potato and put it in the fridge").
MiniHack: A grid-world game where the robot has to navigate mazes, eat food, or use magic wands.

What they found:

The Coach got smarter: The trained Coaches were much better at writing instruction manuals than untrained ones. They successfully taught the Player how to solve new types of puzzles it had never seen before, just by looking at examples from different types of puzzles.
Cross-Game Magic: In a surprising twist, they took a Coach trained only on the grid-world game (MiniHack) and asked it to coach the Player on the house game (ALFWorld). Even though the games were totally different, the Coach still managed to write a manual that helped the Player perform better than a robot with no coaching at all. It learned the skill of learning, not just the specific game rules.
More Turns, Better Results: The more the Coach practiced writing manuals (going through more "meta-turns"), the better the Player got, even beyond the number of practice rounds the Coach had seen during training.

The Toolkit: MetaGym
Finally, the authors released a free software tool called MetaGym. Think of this as a "Lego kit" for researchers. It allows anyone to easily build their own "Coach vs. Player" training environments to test if their AI can learn from experience.

In a Nutshell
This paper proves that AI agents can be trained to stop being "amnesiacs." By using a Coach that watches the Player's experiences and writes better rules for the future, the AI can learn general lessons that apply to new, unseen challenges. It's the difference between a robot that memorizes one map and a robot that learns how to read any map.

Technical Summary: Training Language Agents to Learn from Experience

Problem Statement

Current reflection-based methods for Large Language Model (LLM) agents primarily focus on self-correction within a single task instance. While effective for refining behavior on a specific problem through repeated attempts, these methods fail to distill experience into reusable lessons that improve performance on future, unseen tasks. This limitation is critical in real-world scenarios where tasks cannot be repeated (e.g., chess games or cooking) and where insights from one task instance should ideally generalize to similar, unseen tasks. The central research question addressed is whether LLMs can be trained to generate self-reflections that facilitate cross-task self-improvement, and if so, how to achieve this without relying on human-provided examples.

Methodology

The In-context Training (ICT) Task

The authors introduce the In-context Training (ICT) task, a framework designed to evaluate an LLM's ability to learn from experience for cross-task self-improvement. The framework employs a dual-LLM architecture:

Actor ( $LLM_{act}$ ): An operational agent that interacts with an environment based on a system prompt. Crucially, in each turn of the ICT task, the actor faces a new, unseen batch of tasks and has only one attempt per task.
Reflector ( $LLM_{refl}$ ): A meta-agent that observes the trajectories collected by the actor and generates an improved system prompt ( $sp_i$ ) intended to enhance the actor's performance on the next batch of unseen tasks.

The objective is for the reflector to analyze episodes from a training set ( $E_{train}$ ) and generate a system prompt that maximizes the cumulative reward of the actor on a held-out validation set ( $E_{val}$ ) of unseen task types.

Training Pipeline

To enable the reflector to learn this capability, the authors propose a Reinforcement Learning (RL)-based fine-tuning pipeline:

Dataset Creation: The ICT loop is run multiple times using a base model to generate a dataset of tuples $(sp_i, obs_i, \text{task batch})$ . This captures diverse system prompts and the corresponding meta-observations (interaction trajectories).
Reward Signal: The training reward is derived directly from experience. For a candidate system prompt generated by the reflector, the reward is calculated by replaying the previous batch of tasks using the actor model under the new prompt. This encourages the reflector to identify performance issues in the previous batch and propose fixes that generalize, rather than overfitting to the immediate batch.
Optimization: The reflector is fine-tuned using the GRPO (Group Relative Policy Optimization) algorithm. During training, the reflector samples multiple candidate prompts, which are scored by replaying tasks with a frozen actor. The scores drive advantage estimation and parameter updates.

Infrastructure: MetaGym

The authors introduce MetaGym, a generic Python library for constructing meta-environments from existing Gym-like environments. This tool facilitates the creation of the dual-LLM setup required for ICT, managing task instantiation, parallel execution, and the interaction between actor and reflector.

Key Contributions

A Training Pipeline for Cross-Task Self-Improvement: A method to fine-tune LLMs to generate reflections that improve performance across unseen task instances without relying on human-provided labels or examples.
The In-context Training (ICT) Task: A novel evaluation framework that measures online self-improvement capabilities where tasks are non-repeatable, forcing the model to learn transferable strategies rather than memorizing specific solutions.
MetaGym: An open-source library enabling the construction of meta-environments for research into self-improving language agents.

Experimental Results

The approach was evaluated on two benchmarks: ALFWorld (text-based household tasks) and MiniHack (2D grid-world tasks). In both cases, task types were partitioned into meta-train and meta-test sets to ensure the reflector learned generalizable strategies rather than memorizing specific task instances.

Cross-Task Improvement: Trained reflectors consistently outperformed untrained baselines (Qwen2.5-7B-Instruct) and initial prompts across nearly all held-out task types in both benchmarks. This demonstrates that the ability to learn from experience is itself a learnable meta-skill.
Generalization Beyond Training: In some cases, reflectors trained on MiniHack successfully improved performance on ALFWorld tasks (and vice versa), indicating transferability between substantially different environments.
Meta-Turns and Horizon: Performance continued to improve beyond the number of meta-turns seen during training (turn 4), suggesting the model can recursively refine its own generated prompts.
Batch Size Effects: Larger batch sizes ( $k=5$ ) generally yielded better results, particularly in MiniHack, by providing more diverse experience and reducing the sparsity of reward signals. However, in ALFWorld, smaller batches ( $k=3$ ) were sometimes sufficient, suggesting diminishing returns with larger batches depending on the environment.

Significance and Claims

The paper claims that learning from experience is a learnable capability. By shifting the focus from self-correction within a single task to self-improvement across a family of tasks, the authors provide a pathway for developing general-purpose agents capable of adapting to open-ended environments.

The work emphasizes that:

Reflection mechanisms can be trained to extract reusable lessons, moving beyond the current paradigm where agents must "relearn" lessons for every new task.
This capability can be acquired without human supervision, relying solely on the agent's own interaction data.
The proposed framework (ICT) and tool (MetaGym) offer a standardized way to evaluate and research self-improving agents, potentially shifting the field's attention toward cross-task generalization.

The authors remain modest regarding limitations, noting that maintaining separate actor and reflector models is computationally expensive and that the current reliance on a fixed held-out set for evaluation may not perfectly mirror real-world scenarios where such sets are unavailable. They suggest future work could explore lifelong learning without fixed stopping criteria.