💬 NLP

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

This paper introduces MemCon, a framework that treats memory operations as a learnable Markov Decision Process to dynamically adapt retrieval, injection, and consolidation strategies via a lightweight contextual bandit, thereby significantly improving LLM agent performance and efficiency across diverse benchmarks compared to static, heuristic-based memory systems.

Original authors: Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu, Levina Li, Dong Liu, Xiao Liang, Rui Sun, Yubei Li, Edward Sun, Haozheng Luo, Zhaolu Kang, Aylin Caliskan, Kai-Wei Chang, Ying Nian Wu

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Eric Hanchen Jiang, Zhi Zhang, Yuchen Wu, Levina Li, Dong Liu, Xiao Liang, Rui Sun, Yubei Li, Edward Sun, Haozheng Luo, Zhaolu Kang, Aylin Caliskan, Kai-Wei Chang, Ying Nian Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to play a new video game. You want the robot to get better every time it plays, so you give it a notebook where it can write down what worked and what didn't. This is the world of Large Language Model (LLM) agents: AI programs that can chat, plan, and use tools to solve problems. The key to making them truly smart isn't just the robot's brain; it's the notebook, or memory. If the robot can remember past mistakes, it won't repeat them. If it remembers a clever trick that worked yesterday, it can use it today.

For a long time, scientists thought about this notebook as a static library. They built rules like, "Always look at the last three pages," or "Always search for the most similar story." It's like having a librarian who follows a strict, unchanging script no matter what book you ask for. But real life is messy. Sometimes you need a quick glance at the index; other times, you need to read the whole chapter. Sometimes you need to throw away old notes to make room for new ones. The big question this paper tackles is: Can we teach the robot to manage its own notebook, deciding exactly when to read, when to write, and what to forget, instead of following a rigid script?

The Problem: The Robot with a Broken Librarian

Meet our robot agent. It's trying to solve puzzles, navigate virtual houses, or answer tricky questions. It has a memory system, but until now, that system has been a bit clumsy. Most existing systems use a "fixed pipeline." Imagine a librarian who, no matter what you ask, always pulls out the top five books from the shelf and hands them to you.

If you ask a simple question like "What's the weather?", the librarian might dump five heavy encyclopedias on your desk. That's too much noise! But if you ask a complex question like "How do I fix a broken engine using only tools from 1990?", that same librarian might only hand you a single, irrelevant pamphlet because they are too scared to look deeper. The robot gets stuck, confused by too much information or missing the one crucial clue it needs.

The authors of this paper argue that this "one-size-fits-all" approach is the main reason robots aren't learning faster. They believe the best memory behavior changes depending on the situation:

Early in a game: The robot has no experience yet, so it shouldn't waste time digging through an empty notebook.
When stuck: If the robot is doing the same wrong action over and over, it needs a different kind of search, not the same old one.
When it knows a trick: If the robot has already solved a similar puzzle, it shouldn't search for clues; it should just pull out the "cheat sheet" (a plan) it wrote down before.

The Solution: MEMCON, the Smart Librarian

Enter MEMCON (Memory as a Controlled Process). The authors didn't build a new type of notebook; instead, they built a smart, adaptive librarian that sits on top of any existing memory system.

Think of MEMCON as a tiny, super-fast decision-maker that asks itself three questions before the robot even opens its notebook:

When should I look?
What should I look for?
How much should I grab?

Instead of using a hard-coded rule, MEMCON treats memory management like a game of strategy. It uses a mathematical approach called a Markov Decision Process (imagine a flowchart where every choice leads to a new state). The "player" in this game is the memory controller, and its "moves" include:

Retrieving: Pulling out old notes.
Injecting a Plan: Sliding a pre-written "cheat sheet" into the robot's mind.
Re-retrieving: Trying a new search query because the first one failed.
Consolidating: Merging messy notes into a clean summary.
Forgetting: Throwing away old, useless junk to make space.

The magic is in how MEMCON learns. It doesn't need to be pre-trained on a mountain of data. Instead, it learns on the fly while the robot plays. Every time the robot finishes a task, MEMCON gets a score: "Success!" or "Fail." If the robot succeeded, the memory choices that led to that success get a "high five" (a reward). If it failed, those choices get a "thumbs down."

Over just a few dozen tasks, MEMCON figures out the perfect strategy. It learns that for "early stage" tasks, a shallow search is best. It learns that when the robot is "stuck," it needs to try a completely different search angle. It learns that for complex math problems, it's better to inject a general plan than to search for specific examples.

The Results: Smarter, Faster, and Cheaper

The team tested MEMCON on six different benchmarks, ranging from navigating virtual houses (ALFWorld) to solving science puzzles (ScienceWorld) and answering tricky trivia questions (GAIA). They tried it with three different robot frameworks and three different "brains" (the underlying AI models).

The results were impressive. MEMCON consistently beat the best existing memory systems.

Better Scores: In some tests, MEMCON improved the robot's success rate by up to 15.2 points. For example, on the ALFWorld benchmark, it reached a 67.9% success rate, which was the highest score among all the memory systems tested.
Less Waste: Usually, when you make a system smarter, it uses more computer power. But MEMCON did the opposite. By knowing exactly what to retrieve, it avoided pulling in useless information. This reduced the amount of data the robot had to process by 5% to 20%.
No Extra Cost: Unlike some other systems that ask the AI to "think" about what to remember (which costs extra time and money), MEMCON makes its decisions in milliseconds using a simple lookup table. It adds zero extra calls to the AI model.

Why This Matters

The paper suggests that the future of smart robots isn't just about building bigger brains or bigger notebooks. It's about building better managers for those notebooks.

The authors show that memory shouldn't be a static pipeline where you just dump data in and hope it comes out right. Instead, memory should be a controlled process. By letting the agent learn how to access its own experiences, we can create robots that are not only smarter but also more efficient. They learn faster, make fewer mistakes, and don't get bogged down by their own history.

In short, MEMCON teaches the robot to be the boss of its own memory, ensuring that when it needs a clue, it finds the right one at the right time, without getting lost in a sea of irrelevant notes. It's a small change in how we manage memory, but it seems to make a huge difference in how well these digital agents can learn and grow.

Technical Summary: Memory as a Controlled Process (MEMCON)

Problem Statement

Large Language Model (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. However, existing approaches—from graph-structured memories to reflective insight stores—typically access memory through fixed, hand-designed heuristics. These systems employ static pipelines where a single global retriever is invoked with hard-coded parameters (e.g., top- $k$ , graph hop depth, query templates) regardless of the agent's current state.

The authors argue this static view is a fundamental bottleneck for agentic learning because optimal memory behavior is context-dependent:

Early task stages: Benefit from minimal retrieval as memory is sparse; indiscriminate retrieval dilutes the prompt.
Recurring goal types: Benefit from plan reuse (distilled templates) rather than generic nearest-neighbor lookup.
Stuck agents: Benefit from re-retrieval with alternative queries rather than re-reading failed top- $k$ results.
Long task streams: Require consolidation and pruning to prevent the memory store from becoming overwhelming.

No single fixed heuristic satisfies all these regimes. Furthermore, while some works (e.g., MemGPT) make memory adaptive by using the LLM itself as a controller, this incurs the cost of an additional LLM call per memory operation. The paper posits a need for a learned, lightweight controller that is adaptive without requiring extra LLM calls.

Methodology: MEMCON

The authors introduce MEMCON (Memory as a Controlled Process), a framework that models memory operations as a sequential decision problem.

1. Memory Markov Decision Process (MDP)

MEMCON reformulates memory access as an MDP, $\mathcal{M}_{mem} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)$ :

State ( $\mathcal{S}$ ): A fusion of task-progress signals (goal type, step phase, "stuck" indicator, locations visited) and memory status (size, plan availability, learning phase). States are discretized into a compact hashable key $\phi(s)$ to enable tabular learning.
Actions ( $\mathcal{A}$ ): A finite set of memory operations and their parameters:
- Operations: RETRIEVE, PLANINJECT (prepending a distilled success plan), RE-RETRIEVE (using an alternative query), CONSOLIDATE, FORGET, and NOOP.
- Parameters: top-k, insight-k, and hop depth.
Reward ( $\mathcal{R}$ ): Observed at the end of a task based on environment feedback (success/failure) and efficiency (token/step count). Successful, efficient trajectories receive higher credit.
Transitions ( $\mathcal{T}$ ): The environment transition is opaque to the controller; the system treats the MDP as a contextual bandit with episode-level (Monte-Carlo) feedback.

2. Online Policy Learning

MEMCON learns a policy $\pi: \mathcal{S} \to \mathcal{A}$ online during deployment with zero pretraining and zero additional LLM calls.

Algorithm: A lightweight tabular contextual bandit using Upper Confidence Bound (UCB) exploration.
Warm-Start Priors: Q-values are initialized with interpretable priors (e.g., positive for RETRIEVE and PLANINJECT, negative for NOOP) to accelerate convergence in the first few tasks.
Credit Assignment: Updates use a reverse-discounted Monte-Carlo return. Actions taken closer to the terminal outcome receive stronger credit, allowing the system to learn which memory operations contributed to success or failure.
Persistence: The Q-table is persisted to disk, allowing learning to carry across runs and task streams.

3. Backend-Agnostic Wrapper

MEMCON is not a new memory store but a thin wrapper that intercepts the abstract retrieve and store entry points of any existing memory backend.

It decides how to call the backend (which operation and parameters) based on the learned policy.
It supports two augmented operations:
1. Generalized Plan Injection: Extracts action sequences from successful tasks, replaces instance-specific identifiers with placeholders (e.g., shelf 3 $\to$ [shelf]), and prepends the template to future queries of the same type.
2. Goal Decomposition: A deterministic heuristic for composite tasks (e.g., "put two cellphones") that injects a strategy to handle objects sequentially.

Key Contributions

Formalization: The authors formalize agent memory management as a Memory MDP and introduce a backend-agnostic wrapper that decouples the control policy from the storage backend, allowing any existing or future memory system to inherit adaptive control.
Algorithm: They propose an online contextual bandit policy with UCB exploration, warm-start priors, and reverse-discounted credit assignment. This policy converges within tens of tasks using no pretraining and no extra LLM calls.
Evaluation: Extensive evaluation across 6 benchmarks (interactive decision-making and QA/web/tool-use), 3 agent frameworks, and 3 LLM backbones demonstrates consistent accuracy gains and token savings.

Experimental Results

The authors evaluated MEMCON against 9 strong memory baselines (including vector retrieval, skill libraries, trajectory summarization, insight-based learning, and graph-structured memories) and a no-memory baseline.

Performance: MEMCON consistently achieved the best or near-best task success rates.
- On GPT-4.1-mini, MEMCON attained the top score in 9 of 18 framework $\times$ benchmark cells.
- On Claude Sonnet-4, it was top in 15 of 18 cells.
- On DeepSeek-V3.2, it was top in all 9 interactive cells and 15 of 18 overall.
- Gains were particularly pronounced in interactive decision-making tasks (e.g., ALFWorld, PDDL, ScienceWorld), where MEMCON improved success rates by up to 15.2 points over baselines.
Efficiency: Contrary to the intuition that better performance requires more context, MEMCON reduced token consumption by 5–20% per task while improving accuracy. The learned policy successfully suppressed retrieval when memory was sparse or irrelevant and replaced nearest-neighbor retrieval with plan injection when templates existed.
Generalization: The framework generalized well to QA and web/tool-use benchmarks (TriviaQA, WebWalkerQA, GAIA), showing that adaptive control is beneficial beyond purely interactive environments.
Ablation: Component analysis revealed that the learned UCB controller was the largest contributor to performance gains (accounting for the majority of the improvement over static backends), while the augmented operations (plan injection, goal decomposition) provided additional, though optional, benefits in specific scenarios.

Significance and Claims

The paper claims that effective long-term memory for LLM agents depends not only on what is stored but on learning how memory should be accessed and managed over time.

By treating memory operations as a controlled decision process rather than a static pipeline, MEMCON demonstrates that:

Adaptivity is crucial: A single fixed heuristic cannot optimize memory access across varying task phases, memory states, and agent progress.
Lightweight control is viable: A learned controller can be implemented via a simple tabular bandit with UCB, avoiding the computational overhead of using the LLM itself as a controller (as in MemGPT).
Decoupling is beneficial: Separating the memory control policy from the storage backend allows for modular improvements where any memory system can instantly gain adaptive capabilities without re-engineering its storage layer.

The authors conclude that replacing fixed memory pipelines with lightweight learned controllers yields consistent, framework- and backbone-agnostic accuracy gains while simultaneously reducing computational costs.