🤖 AI

Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents

This paper demonstrates that frozen-weights AI agents can achieve significant continual learning improvements by pairing their static models with an external memory system that distills deployment feedback into retrievable natural-language rules, thereby enabling them to solve previously intractable tasks without model retraining.

Original authors: Valentin Tablan, Scott Taylor, Kristoffer Bernhem

Published 2026-07-27

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Valentin Tablan, Scott Taylor, Kristoffer Bernhem

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where every time you tried to solve a puzzle, you forgot the rules the moment the timer stopped. If you solved a tricky math problem today, you'd start from scratch tomorrow, even if the problem was exactly the same. This is the current reality for many artificial intelligence (AI) agents. They are built on "frozen" brains—powerful models that are set in stone when they are launched. They can't learn from their mistakes in real-time because changing their brain's wiring is expensive, risky, and slow. However, in the real world, these agents are constantly getting feedback. A customer might say, "That worked!" or a human might fix a mistake and say, "Here is the right way." The big question scientists are asking is: Can we teach these frozen-brain agents to get smarter just by reading a "cheat sheet" of their own past experiences, without ever actually changing their brain?

This paper, titled "Learning on the Job," explores a clever workaround. Instead of trying to rewire the AI's brain, the researchers gave the agent an external notebook—a memory system. When the agent finishes a task, it looks at the result (did it win or lose?) and writes a short, natural-language rule into this notebook. For example, "If the customer asks for a refund, check the date first." Later, when the agent faces a similar situation, it checks its notebook before acting. The study tested this on a banking simulation where the AI had to navigate complex rules to help customers. They found that by simply reading their own notes, the agents got significantly better at their jobs. An agent that learned from just a "win/loss" signal improved its success rate by 1.6 times, while one that learned from detailed corrections improved by 2.6 times. Even cooler, the notes written by one type of AI were useful to a completely different type of AI, proving that this "experience" could be shared like a library book.

The Problem: The AI Amnesia

Think of today's most advanced AI agents as incredibly talented but incredibly forgetful interns. You hire them, they are given a massive library of policy books (about 700 documents in this study) to reference, and they try to help a customer. If they solve a hard problem, great! But the moment the conversation ends, their short-term memory wipes clean. If that same difficult customer calls back tomorrow, the agent has no idea what happened yesterday. It has to start from zero.

The reason for this amnesia is architectural. These agents run on models with "frozen weights." Imagine the model's brain as a giant, intricate sculpture made of clay. Once the sculpture is baked and hardened (deployed), you can't chip away or add new clay without melting the whole thing down and starting over. Melting it down is dangerous (it might forget everything it knew before) and expensive. So, in the real world, these agents stay frozen. They are as capable on their last day of work as they were on their first.

The Solution: The External Notebook

The researchers asked: What if the agent didn't need to change its brain to learn? What if it just needed a notebook?

They paired the frozen AI with a system called Spark. Think of Spark as a shared, magical library where the agent can write and read notes. When an agent finishes a task, it doesn't just throw the experience away. Instead, it reflects on what happened and writes a "rule" into the library.

The "Experience" Mode: The agent gets a simple thumbs-up or thumbs-down (a one-bit verdict). It has to figure out the lesson itself. "I tried to do X and failed. I should avoid X next time."
The "Instruction" Mode: The agent gets the thumbs-down plus the correct answer. "I tried X and failed. The right way is Y."

The agent writes these lessons as simple, natural language rules, like "When a customer asks for a transfer, always check the daily limit first." These rules are stored in the Spark library. The next time the agent faces a customer, it searches the library for relevant rules before it acts.

The Experiment: Banking on Memory

To test this, the team used a tough banking benchmark called τ-bench. Imagine a video game where the AI plays a bank teller. The "customers" are simulated by other AIs, and they have hidden goals and tricky requests. The agent has to figure out the bank's rules by searching through a huge pile of policy documents. It's hard; even the best AI models only solve about 25% of these tasks on their first try.

The researchers ran the experiment with two different AI models:

Mistral Large: A powerful, open-source model that companies could run on their own servers.
Claude Sonnet 5: A top-tier, frontier model.

They tested three scenarios for each model:

The Baseline: The agent has the policy library but no memory. It forgets everything between attempts.
Experience: The agent has the library and a memory notebook, but only learns from a simple "win/loss" signal.
Instruction: The agent has the library, a memory notebook, and learns from the "win/loss" signal plus the correct solution after a failure.

The Results: From Zero to Hero

The results were striking. The agents that used the memory notebook got much better at their jobs, and they did it without changing a single neuron in their frozen brains.

The Power of Corrections: For the Mistral Large model, the "Instruction" group (learning from corrections) solved 2.6 times more tasks on the first try compared to the baseline. They managed to solve 22 out of 84 tasks that the baseline agent never solved, no matter how many times it tried.
The Power of Experience: Even just learning from a simple "win/loss" signal helped. The "Experience" group improved by 1.6 times over the baseline.
The Learning Curve: In the beginning, all agents were equally bad because the notebooks were empty. But as the trials went on, the agents with notebooks got smarter. The baseline agent stayed flat, while the learning agents climbed higher with every attempt.
Cross-Model Magic: The researchers did something really cool. They took the notebook built by the Mistral Large model and gave it to the Claude Sonnet 5 model to read (and vice versa). Even though they were different models, the notes were useful! The Mistral model, when reading Claude's notes, improved its success rate significantly. This proves that the knowledge stored in the notebook wasn't just "Mistral's style"; it was general, useful knowledge that any agent could use.

Why This Matters

This paper suggests a new way to make AI smarter without the huge cost and risk of retraining. Instead of trying to melt and reshape the AI's brain, we can just give it a better way to remember.

The study also ruled out a few ideas. For instance, they checked if the agents were just "caching" answers (memorizing the exact solution to a specific puzzle). They found that wasn't the case. The agents were learning rules (like "check the date") that helped them solve new variations of the problem. They also showed that this learning didn't make the agents worse at things they were already good at; they kept their old skills while gaining new ones.

In short, this research shows that for frozen AI agents, the feedback they get every day—whether it's a simple "good job" or a detailed correction—is enough to turn them into lifelong learners, provided they have a place to write down what they've learned. It turns a single agent's experience into a shared organizational asset, making the whole system smarter, one note at a time.

Technical Summary: Learning on the Job for Frozen-Weights Agents

Problem Statement

Current AI agent deployments rely on foundation models with frozen weights, meaning agents discard nearly all learning opportunities after each episode. When a difficult request recurs, the agent starts from zero. While weight updates offer a classical route to continual learning, they incur prohibitive costs regarding infrastructure, the risk of catastrophic forgetting, and the need for fresh safety evaluations. Consequently, deployed agents often remain as capable on their final day as on their first.

Real-world deployments typically generate supervision signals—such as outcome verdicts (success/failure) or post-hoc corrections (verified resolutions)—yet these signals are rarely utilized for learning. This paper investigates whether these ordinary feedback signals are sufficient to enable continual learning without modifying model weights, by pairing the frozen agent with an external memory system.

Methodology

The Spark Memory System

The study utilizes Spark, a shared memory service for AI agents.

Unit of Knowledge: Insights are stored as compact natural-language statements (rules, procedures, warnings) with a title, content, and a retrieval query phrased as a question a future agent would ask.
Lifecycle: Contributions undergo an autonomous curation pipeline involving safety filtering, quality gates, and deduplication. The system supports continuous pruning and synthesis of new insights from clusters of related ones.
Interaction: Agents interact via four calls: search, create_memory, enrich_memory, and share_feedback. The system operates on a read-contribute-feedback cycle, where feedback updates trust estimates to influence retrieval ranking.

Experimental Setup

The experiments are conducted on the $\tau$ -Banking domain of $\tau$ -bench, a knowledge-intensive benchmark involving multi-turn conversations with simulated customers.

Task Difficulty: Agents must discover business rules and tools by searching a corpus of ~700 policy documents using sparse BM25 retrieval. The benchmark is deterministic, with success defined by the final database state matching a ground truth.
Models: Two models span the deployment spectrum: Mistral Large (open-weights, self-hostable) and Claude Sonnet 5 (frontier model).
Protocol: Each task runs $k=4$ independent trials. The memory store starts empty and remains active across trials for a single task, allowing lessons from trial $t$ to influence trial $t+1$ .

Conditions

Three conditions are compared:

No-memory Baseline: A static RAG control retrieving over the complete policy corpus with no inter-trial memory.
Experience: The agent learns from a single-bit outcome verdict (success/failure). On failure, it must distill its own lessons (avoidance rules) without access to the correct solution.
Instruction: The agent learns from the verdict plus a verified action sequence (correction) delivered after a failed episode. This allows the agent to write the correct rule into memory.

Metrics

$pass^k$ : The standard benchmark metric requiring a task to be solved in all $k$ trials.
Per-trial success rate: The pooled success rate as a function of trial index.
Hold rate: The probability that a task passed at trial $t$ is passed again at $t+1$ , measuring knowledge retention.
Conversion: The rate at which tasks in the "floor stratum" (tasks the baseline never solves) are solved by the learning conditions.

Key Results

Continual Learning Performance

Mistral Large:
- Instruction lifted single-trial success ( $pass^1$ ) to 0.170, which is 2.6 $\times$ the baseline (0.064) and 1.6 $\times$ the Experience condition (0.103).
- Experience alone lifted $pass^1$ to 0.103 (1.6 $\times$ baseline).
- Conversion: Instruction converted 22 of the 84 tasks the baseline never solved. Experience converted 5.
- Retention: Experience held banked solutions at a rate of 0.88, while Instruction held at 0.65.
Claude Sonnet 5:
- Replicated the trend on a stronger model. Instruction lifted $pass^1$ from 0.248 (baseline) to 0.397.
- Converted 28 of the 57 tasks the baseline never solved.
- The learning curve showed a monotonic rise in success rates across trials for learning conditions, whereas the baseline remained flat.

Cross-Model Transfer

The study tested whether memory built by one model could benefit another by freezing stores and having the other model read them (read-only).

Mistral Large reading the Claude Sonnet 5-built store improved $pass^1$ from 0.064 to 0.289 (+0.224).
Claude Sonnet 5 reading the Mistral Large-built store improved $pass^1$ from 0.248 to 0.314 (+0.066).
Significance: The transfer was effective in both directions, including "uphill" transfer where the weaker model's rules helped the stronger model. Conditional on a task being transferable (unsolved by the consumer but solved by the producer), conversion rates were approximately 31% in both directions.

Significance and Claims

The paper claims three primary contributions:

Sufficiency of Post-Episode Feedback: The study demonstrates that ordinary deployment feedback (outcome verdicts and corrections) is a sufficient signal for continual learning. An agent can lift its performance significantly (up to 2.6 $\times$ ) without weight updates, provided it is paired with an external memory that distills episodes into retrievable rules.
Transferability of Accumulated Memory: The accumulated memory stores are portable across different models. A frozen store built by one model can improve the performance of a different model, turning individual agent experience into organizational knowledge that is model-agnostic.
Reusable Measurement Protocol: The paper introduces a protocol for measuring continual learning in frozen-weight agents, distinguishing between task success, reliability across trials, and knowledge retention (hold rate), released alongside the harness and data.

The authors emphasize that the results address the specific regime where knowledge the model lacks is decisive. The learning effect appears between trials, driven solely by the memory store, and the accumulated knowledge transfers effectively, suggesting that external memory can serve as a viable alternative to weight updates for many enterprise deployments. The paper explicitly notes that generalization to unseen task types was not measured; the focus is on recurring situations in deployment.