🤖 machine learning

Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMs

SEAM is a lightweight, plug-in module that stores experience within its own parameters and generates structured, utility-optimized guidance for frozen LLMs in a single forward pass, improving reasoning performance without the latency or noise of traditional retrieval-based methods.

Original authors: Xuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuancheng Li, Haitao Li, Yujia Zhou, Yiqun Liu, Qingyao Ai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a world-class chef (the LLM Executor). You are incredibly skilled, but you have a bit of a problem: you have no long-term memory. Every time a customer walks in and asks for a dish, you have to start from scratch. Even if you just made a complicated soufflé five minutes ago and realized you added too much salt, the moment the next order comes in, you forget that mistake and might do it all over again.

Currently, most people try to help chefs by giving them a massive, dusty cookbook (this is RAG/Retrieval). When a chef needs help, they have to stop, flip through thousands of pages, find a recipe that looks similar, and try to follow it. This is slow, and sometimes the cookbook gives them a recipe that is "sort of" related but actually makes the dish worse.

This paper introduces SEAM, and it changes the game.

The Concept: The "Personal Sous-Chef"

Instead of a giant, slow cookbook, the researchers gave the chef a tiny, lightning-fast Sous-Chef (SEAM).

This Sous-Chef doesn't carry a library of books. Instead, the Sous-Chef has "learned" the chef's specific habits, strengths, and common mistakes through practice. When a new order comes in, the Sous-Chef doesn't go searching through a library; they just whisper a quick, structured note into the chef's ear.

This note isn't just a recipe. It’s a "Cheat Sheet" that contains three things:

The "Heads Up": "Hey, this dish is tricky because the heat is very sensitive." (Problem Analysis)
The "Pro-Tip": "Remember, last time we did this, we forgot to preheat the pan." (Experience Highlights)
The "Game Plan": "First chop, then sauté, then simmer." (Reference Plan)

How the Sous-Chef Learns (The Training)

The researchers didn't just tell the Sous-Chef what to say. They used a method called GRPO, which is like a high-intensity training camp.

It works like this:

Trial and Error: The Sous-Chef suggests three different "cheat sheets" for a dish.
The Test: The Chef tries to cook the dish using each cheat sheet.
The Feedback: If the dish tastes amazing, the Sous-Chef gets a "gold star." If the dish is a disaster, the Sous-Chef gets a "red card."
Evolution: Over time, the Sous-Chef learns exactly which kind of advice actually helps this specific chef succeed.

Why is this better?

It’s Faster: There’s no searching through a library. The Sous-Chef just speaks up instantly.
It’s Smarter: The advice isn't based on "what looks similar," but on "what actually works." It’s optimized for utility, not just similarity.
It’s Personalized: The Sous-Chef learns the specific quirks of the Chef. If the Chef is great at chopping but bad at timing, the Sous-Chef focuses on the timing.
It’s Lightweight: You don't have to retrain the master Chef (which is expensive and difficult). You only train the tiny Sous-Chef.

The Bottom Line

In the world of AI, we usually try to make the "brain" (the LLM) bigger and bigger. This paper suggests a different path: Keep the brain as it is, but give it a smart, specialized assistant that turns past mistakes into instant, actionable wisdom.

Technical Summary: Beyond Experience Retrieval: Learning to Generate Utility-Optimized Structured Experience for Frozen LLMs

1. Problem Statement

Large Language Models (LLMs) are inherently static; when faced with new problems, they often reason from scratch, repeating previous mistakes or re-exploring inefficient solution paths. Current methods to mitigate this typically rely on Retrieval-Augmented Generation (RAG), where an external library of past experiences is searched via similarity matching.

The authors identify three critical flaws in the RAG-based approach:

Similarity $\neq$ Utility: Retrieval is optimized for semantic similarity, which does not guarantee that the retrieved information is actually helpful for solving the current task. It can introduce noise or miss critical constraints.
Inference Overhead: Maintaining, indexing, and querying an external database adds significant latency and computational cost.
Static Templates: RAG often relies on hand-crafted templates that are difficult to maintain as tasks evolve.

2. Methodology: SEAM (Structured Experience Adapter Module)

The paper proposes SEAM, a lightweight, trainable, and executor-specific plug-in that replaces external retrieval with parametric experience. Instead of searching a database, SEAM encodes experience within its own parameters and generates a tailored "experience entry" in a single forward pass.

A. Architecture & Schema
SEAM generates a structured, schema-constrained prompt consisting of three components to guide a frozen executor LLM:

Problem Analysis: A diagnostic assessment of the instance's difficulty and potential failure modes.
Experience Highlights: Actionable insights and heuristics distilled from prior successful rollouts.
Reference Plan: A procedural, step-by-step workflow (not the final answer) to guide the reasoning process.

B. Training Pipeline (Forward Learning)
SEAM is trained using a three-step process that optimizes for utility (task success) rather than similarity:

Forward Exploration: For a given instance, SEAM samples $K$ candidate experience entries.
Rollout-based Evaluation: The frozen executor attempts to solve the problem conditioned on each candidate. Each attempt is scored based on whether the executor reaches the correct answer and whether the experience entry follows the required schema.
Parametric Library Evolution (GRPO): SEAM is updated using Group Relative Policy Optimization (GRPO). It computes the advantage of each candidate relative to the group average and updates its parameters to increase the likelihood of generating high-utility guidance.

C. Deployment-time Evolution (Optional)
Post-deployment, SEAM can undergo Supervised Fine-Tuning (SFT) using a buffer of successful (instance, experience) pairs logged during real-world usage, allowing it to adapt to new distributions.

3. Key Contributions

Shift from Retrieval to Generation: Moves experience management from an external, similarity-based search to an internal, utility-optimized generative process.
Executor-Specific Optimization: Recognizes that different LLMs have different failure modes; SEAM is trained to provide guidance specifically tailored to the strengths and weaknesses of a particular frozen executor.
Decoupled Training: By keeping the executor frozen, SEAM allows for efficient, modular adaptation without the massive cost of fine-tuning large-scale models.
Structured Guidance: Introduces a formal schema that ensures the guidance is diagnostic, prescriptive, and procedural.

4. Experimental Results

The authors evaluated SEAM on mathematical reasoning benchmarks (GSM8K, MATH, AIME24, AIME25) using various executors (Qwen3, DeepSeek-R1).

Superior Accuracy: SEAM consistently outperformed the original models, RAG-based baselines (MEM-0, Memento, etc.), and even direct GRPO training of the executor itself.
Difficulty Scaling: The performance gains were most significant on highly challenging tasks (e.g., AIME), suggesting that structured experience is most valuable when reasoning becomes complex.
Efficiency: SEAM provides a much lower "Time-to-Correct" (TTC) than RAG methods because it avoids the latency of external database lookups.
Generalization: While SEAM is optimized for specific executors, it demonstrated surprising cross-domain generalization (e.g., math-trained SEAM helping in coding/QA) and some degree of cross-executor transferability.
Scaling Laws: Performance improves monotonically as the amount of experience used during training increases.

5. Significance

This work introduces a new paradigm for agentic memory. By treating experience as a learned, generative policy rather than a static database, SEAM provides a more efficient and effective way for LLMs to "learn from doing." It demonstrates that a small, specialized module can significantly amplify the reasoning capabilities of much larger, frozen models, offering a scalable path toward autonomous agents that evolve through interaction.