🤖 AI

Managing Procedural Memory in LLM Agents: Control, Adaptation, and Evaluation

This paper introduces AFTER, a comprehensive benchmark for evaluating procedural memory in LLM agents across diverse enterprise tasks, demonstrating that refined skills significantly boost performance and enable effective cross-model and cross-role transfer while highlighting the varying degrees of generalizability among different skills.

Original authors: Julia Belikova, Rauf Parchiev, Evgeny Egorov, Grigorii Davydenko, Gleb Gusev, Andrey Savchenko, Maksim Makarenko

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Julia Belikova, Rauf Parchiev, Evgeny Egorov, Grigorii Davydenko, Gleb Gusev, Andrey Savchenko, Maksim Makarenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new employee to work in a busy office. You have two ways to train them:

The "Blank Slate" Approach: You give them the task and hope they figure it out using their general intelligence.
The "Procedural Memory" Approach: You give them a specific, reusable "cheat sheet" or "standard operating procedure" (SOP) based on how similar tasks were solved in the past.

This paper, titled "Managing Procedural Memory in LLM Agents," is about testing how good these "cheat sheets" really are for AI workers (LLM agents) and figuring out how to make them better.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Over-Specialized Intern"

The authors noticed that while AI agents are getting smarter, they often struggle with repetitive office tasks (like processing documents, managing databases, or writing code).

They found a tricky problem: If you train an AI on a very specific set of tasks (like "how to process invoices for this specific company"), the AI becomes an expert at that one thing but fails miserably if you ask it to do a slightly different job or work for a different department.

The Analogy: Imagine a chef who learns to make a perfect pizza only in a specific kitchen with specific ovens. If you move that chef to a new kitchen with a different oven, they might burn the pizza because they memorized the specific oven's quirks rather than learning the general skill of "making pizza." This is called overfitting.

2. The Solution: The "AFTER" Benchmark

To solve this, the researchers built a giant testing ground called AFTER.

What it is: A collection of 382 realistic office tasks (like a data engineer fixing a pipeline or a project manager summarizing a report).
The Twist: They didn't just test if the AI could do the task; they tested if the AI could take a "skill" learned in one situation and use it in a different situation (a different role, a different type of task, or even a different AI model).

3. Key Findings: What Works and What Doesn't

A. "Cheat Sheets" Do Help (But Not Always)

When they gave the AI agents these procedural "cheat sheets" (skills), the AI got better at its job.

The Result: On average, adding these skills improved the AI's success rate by about 2.8 points.
The "Refinement" Trick: If they took a basic cheat sheet and let the AI look at its mistakes once to improve the sheet, the success rate jumped another 5.2 points. It's like giving the intern a quick "debrief" after their first day to fix their notes.

B. The "Diverse Experience" Secret

This is the most important discovery. The researchers asked: Where should the cheat sheet come from?

Scenario A: The cheat sheet is written based on the AI's own past attempts (Narrow experience).
Scenario B: The cheat sheet is written by combining attempts from many different AI models (Diverse experience).

The Winner: Scenario B.

The Analogy: If you want to learn how to drive, it's better to read a manual written by 100 different drivers (some fast, some slow, some in rain, some in snow) than to just memorize the exact path your one friend took to the grocery store.
The Data: Skills built from "diverse" experiences (many different AI models) achieved 73.1% accuracy when tested on new models. Skills built from just one model only got about 36–59%. Surprisingly, even "weaker" AI models provided useful lessons when mixed together!

C. The "Role Trap"

The study found that skills can become too specialized to a specific job title.

The Analogy: A "Document Processing" skill learned by a Project Manager (who uses PDFs to summarize meetings) is useless to a Data Scientist (who uses PDFs to extract raw numbers). If you give the Project Manager's cheat sheet to the Data Scientist, the Data Scientist actually performs worse than if they had no cheat sheet at all.
The Lesson: You cannot just copy-paste a skill from one department to another. The context matters.

D. Saving Money (Tokens)

Finally, using these evolved "cheat sheets" saves money.

The Result: Because the AI already knows how to do the task from the cheat sheet, it doesn't need to "think" as hard or ask as many questions. This reduced the amount of "computing power" (tokens) needed by up to 62% in some cases. It's like taking a shortcut that saves you gas money.

Summary

The paper concludes that the future of AI agents isn't just about making them "smarter" in general. It's about teaching them to build reusable, adaptable skills.

Don't just memorize one specific path (that leads to failure in new situations).
Do learn from a wide variety of experiences (different models, different roles).
Do be careful not to mix up skills between different job roles, as they might conflict.

The goal is to move from AI that just "guesses" to AI that has a reliable, evolving library of "how-to" guides that work in the real world.

Technical Summary: Managing Procedural Memory in LLM Agents

Problem Statement

While Large Language Model (LLM) agents are increasingly deployed in industrial workflows to handle recurring tasks (e.g., document processing, database querying, infrastructure configuration), the mechanisms for improving their performance over time remain poorly understood. Current approaches often conflate local improvements within a single environment with true transferability. Existing benchmarks typically evaluate static, expert-curated skills or focus on task completion without isolating the "procedural memory" component—the reusable layer of instructions and strategies distilled from prior trajectories.

A critical gap exists in understanding how procedural knowledge transfers across different contexts. Specifically, it is unclear whether skills evolved from narrow, single-context experiences generalize to new tasks, roles, or model backbones, or if they merely overfit to their source environment. The field lacks a controlled setting to determine when experience produces reusable procedural structures versus when it results in source-context overfitting.

Methodology

The AFTER Benchmark

To address these gaps, the authors introduce AFTER (Agent Framework for Transfer Evaluation and Refinement), a benchmark designed to evaluate procedural skill transfer.

Scope: Contains 382 realistic enterprise tasks spanning six professional roles (Data Engineers, Data Scientists, Generative AI Engineers, Infrastructure Engineers, Project Managers, and Software Engineers) and 22 procedural skills.
Structure: Tasks are categorized into single-skill and multi-skill workflows. Skills are defined as self-contained SKILL.md artifacts (procedural memory units) covering five capability areas: document processing, data operations, ML/AI, infrastructure, and software engineering.
Evaluation Splits: The benchmark utilizes controlled splits to measure:
- Specificity: In-context gain (improvement on the source task set).
- Generality: Performance under distribution shifts, including cross-task, cross-role, and cross-model transfer.
Task Sources: Tasks are derived from existing benchmarks (e.g., SWE-bench, SkillsBench) and newly authored scenarios, all verified via pytest suites to ensure robustness and prevent oracle leakage.

Evaluation Framework: EVOLUTION

The authors employ EVOLUTION, a lightweight harness that standardizes the procedural memory lifecycle:

Collection: Gathering execution traces ( $D$ ) under a specific skill configuration ( $\Sigma$ ).
Diagnosis & Revision: A reflector model ( $\rho$ ) inspects traces, identifies failure modes, and proposes a revised skill body.
Update: An update operator $U$ maps experience to a new configuration: $\Sigma' = U(\Sigma, D)$ .
Measurement: The system evaluates the updated skill on both source (specificity) and held-out (generality) distributions.

The study compares static skill valuations (handcrafted vs. LLM-generated) against dynamic evolution strategies using narrow (single trace) and diverse (multi-model, multi-trace) experience pools.

Key Contributions

AFTER Benchmark: The first benchmark explicitly structured to disentangle local improvement from cross-context transfer in LLM agents. It uniquely combines role-driven task distributions with controlled transfer splits.
Empirical Evidence on Transfer: A comprehensive evaluation of how procedural memory evolves. The study demonstrates that while procedural memory yields consistent gains, its effectiveness is highly dependent on the diversity of the training experience and the target context.
Refinement of Procedural Memory: The introduction of a controlled protocol to evolve skills from execution traces, moving beyond static prompt engineering to dynamic, data-driven skill optimization.

Results

Static Skill Valuation

Baseline Gains: Static procedural skills improve full-pass accuracy ( $M2$ ) by an average of +2.8 points across models.
Model Dependency: Weaker models benefit more consistently from procedural skills than frontier models. For instance, Gemma 4 E4B saw a +14.2 point gain on GenAI tasks, whereas GPT 5.4 saw only +3.1.
Source Quality: LLM-generated skills often outperform handcrafted expert skills, suggesting automated authoring can match or exceed human-derived procedural knowledge.

Skill Evolution and Refinement

Single-Round Refinement: A single round of LLM-guided refinement improves aggregate performance by 3.7–6.7 points across model scales.
Diversity vs. Narrowness:
- Narrow Experience: Skills evolved from a single trace often exhibit "source-context overfitting," improving specificity (train performance) while degrading generality (test performance).
- Diverse Experience: Skills evolved from diverse, multi-model traces achieve 73.1% cross-model test accuracy, outperforming the best single-model trace source by +13.7 points.

Transfer Limitations

Cross-Model Generalization: Skills evolved from diverse traces significantly outperform those evolved from a single source model, even when the source models are weaker. This suggests procedural knowledge benefits from imperfect executions and diverse failure modes.
Cross-Role Specialization: Skills often fail to transfer across roles. For example, a PDF skill evolved for Project Managers (executive summarization) loses effectiveness when applied to Data Scientists (data ingestion), resulting in performance drops of -4.8 to -7.5 points. This indicates natural specialization to role-specific workflows.
Efficiency: Evolved skills reduce inference costs by front-loading knowledge. In a Kafka Lag Anomaly Detection task, evolved skills reduced token usage by 62% (326k tokens) for Claude compared to handcrafted skills.

Significance and Claims

The paper claims that the central challenge of procedural memory is not merely storing more experience, but extracting procedural structures that remain useful beyond the environment in which they were learned.

The authors position AFTER as a practical tool for building, evaluating, and deploying procedural memory systems in production. The results suggest that:

Diversity is key: To achieve generalizable skills, memory systems must aggregate traces from diverse models and contexts rather than relying on a single "expert" trajectory.
Context matters: Procedural memory is not universally transferable; it naturally specializes to specific roles and workflows. Deployments must account for this specialization rather than assuming a "one-size-fits-all" skill library.
Evolution over Statics: Moving from static prompts to evolving, trace-refined skills offers a viable path to improving agent reliability and efficiency in industrial settings.

The authors maintain modesty regarding their scope, noting that the benchmark focuses on technology-sector roles and procedural workflows, excluding open-ended creative or conversational tasks. They also acknowledge that real-world deployments may involve larger trace pools than those tested, leaving the relationship between trace volume and transfer quality as an open question.