Technical Summary: AdMem – Advanced Memory for Task-solving Agents

Problem Statement

Large Language Models (LLMs) have demonstrated significant progress in reasoning and tool use but remain limited in long-horizon tasks that require remembering, organizing, and reusing knowledge across sessions. While prior research has addressed memory challenges through architectural changes (e.g., differentiable attention) or symbolic/textual stores (e.g., retrieval-augmented generation, context paging), existing approaches often suffer from specific limitations:

1. Focus on Factual/Episodic Data: Many systems prioritize storing factual information or compressing history, neglecting procedural memory (decision-making guidance).
2. Ineffective Procedural Memory: Recent procedural memory works often reduce to replaying successful trajectories without addressing failure cases or providing granular feedback for critical steps where models typically fail.
3. Lack of Online Scalability: Most procedural memory systems operate in offline training-inference settings with sparse, binary task-level feedback. They lack mechanisms for automatic memory evaluation, active erasing, reranking, and adaptive recall in online, lifelong environments.
4. Theory-Practice Gap: While theoretical frameworks propose ideal agents with short-term/long-term, semantic/episodic/procedural memory and feedback loops, practical implementations often omit critical components like reward-driven updates, belief modeling, and credit assignment.

Methodology: The AdMem Framework

The authors introduce AdMem, a unified, automatic memory framework designed for lifelong task-solving agents. It integrates semantic, episodic, and procedural memory within a bi-level design (short-term and long-term) and utilizes a multi-agent architecture comprising an Actor, a Critic, and a Memory Agent.

1. Multi-Agent Architecture

• Actor Agent: An LLM-based agent that interacts with the environment. It maintains a Short-Term Memory (STM) state updated per turn and cleared at the end of a task. The STM is encoded into long-term memory. The Actor uses memory tools to generate plans, thoughts, and sub-goals.
• Critic Agent: An LLM-based agent responsible for procedural memory generation. It receives "half-baked" memory entries from the Actor (context, action, expected outcome) and actual observations. It compares expected vs. actual outcomes to:
  • Summarize context and actions.
  • Propose better alternatives.
  • Assign a reward signal ($r_t \in \{0, 1\}$) and reflection.
  • Generate a procedural memory entry to guide future decisions.
• Long-Term Memory (LTM) Agent: A retrieval-based agent managing the persistent store $M$. It stores three types of memory:
  • Semantic Memory ($M_S$): Facts and general knowledge about the environment/tasks.
  • Episodic Memory ($M_E$): Summaries of past interaction events.
  • Procedural Memory ($M_P$): Decision-making guidance extracted from past actions, including expected outcomes and reward annotations.

2. Memory Management and Retrieval

• Bi-Level Architecture:
  • Short-Term Memory (STM): Uses a stack-based structure inspired by computer program execution. It maintains the current sub-goal and plan, compressing turns outside the current sub-goal to manage context length efficiently.
  • Long-Term Memory (LTM): Managed via reward-based evaluation, merging, and pruning.
• Procedural Memory Retrieval & Evaluation:
  • The system models the effectiveness of a procedural memory entry $m$ using an adaptive parameter $v_m \in [0, 1]$.
  • It assumes task success is a binary stochastic process influenced by whether a retrieved memory helps.
  • An Expectation-Maximization (EM) algorithm updates $v_m$ based on observed rewards.
  • Retrieval scores combine context similarity (dense embeddings) and estimated effectiveness ($v_m$).
• Pruning and Merging:
  • Memories retrieved below a frequency threshold $\epsilon$ are deleted.
  • Memories constantly retrieved together (frequency $> 1-\epsilon$) are merged to improve retrieval efficiency.

3. Execution Pipeline

The Actor generates actions and explicit expectations for outcomes. The Critic caches these scenarios. Once the Actor receives the actual observation, the Critic evaluates the outcome against the expectation, generates a reward, and updates the memory store. This allows the system to learn from both successes and failures, addressing the credit assignment problem in multi-step tasks.

Key Contributions

1. Unified Memory Framework: AdMem is the first to integrate semantic, episodic, and procedural memory in a unified, automatic system with a bi-level (short/long-term) design.
2. Procedural Memory with Feedback: Unlike prior works that merely store successful workflows, AdMem uses a Critic agent to annotate procedural memories with reward signals and reflections, enabling the agent to learn from failure cases and critical steps.
3. Adaptive Memory Management: The framework introduces a reward-driven mechanism for evaluating, merging, and pruning procedural memories, ensuring scalability and continual improvement in online settings.
4. Multi-Agent Coordination: The separation of concerns (Actor for execution, Critic for evaluation, Memory Agent for storage/retrieval) enables automatic memory generation and adaptive retrieval without manual policy design.

Experimental Results

The authors evaluated AdMem on the AgentBoard environment, simulating a lifelong, streaming multi-task setting across diverse domains (Embodied AI, Text Games, Web Research, Tool Calling). The backbone LLM was Claude Haiku 4.5.

• Baselines: Compared against ReAct (no cross-task memory) and AWM (Agent Workflow Memory, a procedural memory baseline).
• Performance:
  • AdMem significantly outperformed baselines in most domains, particularly in AlfWorld (63.4% completion vs. 47.0% for AWM) and Tool-query (94.7% vs. 55.0%).
  • In domains where memory transfer is less beneficial, AdMem maintained performance on par with or slightly better than baselines, avoiding the "confusion" often seen in naive memory implementations.
• Ablation Studies:
  • Short-term planning alone improved progress but did not scale with horizon length.
  • Long-term memory alone caused early performance degradation ("burn-in") without the support of short-term planning tools.
  • Full AdMem showed the best results, with performance improving over multiple epochs as the agent accumulated and refined its memory.
• Efficiency: While the multi-agent pipeline incurs extra LLM calls (up to 3x in simple tasks), the parallel execution of agents minimizes latency, and the system prioritizes robustness and learning capability over minimal per-step cost.

Significance and Claims

The paper claims that AdMem bridges the gap between theoretical cognitive architectures and practical LLM agent implementations. By moving beyond simple factual storage to adaptive procedural learning, the framework enables agents to:

• Self-evolve through experience in online environments.
• Customize behavior based on user and task requirements.
• Improve robustness in long-horizon, multi-turn tasks where standard LLMs struggle with context limits and credit assignment.

The authors emphasize that comprehensive, adaptive memory is crucial for advancing LLM-based agents, particularly in scenarios requiring sustained interaction and iterative improvement. The work highlights that successful agent design requires not just storing information, but actively evaluating, refining, and retrieving it based on reward signals.