💬 NLP

Dual-Layer Agentic Memory with Fast Write Routing and Slow Consolidation

Inspired by neuroscience's Complementary Learning Systems theory, this paper proposes Dual-Layer Agentic Memory, a framework that optimizes agent memory by using a cost-aware model cascade to selectively route and prune incoming information for external storage, followed by periodic parametric consolidation to internalize high-value knowledge, thereby maintaining high retrieval accuracy while significantly reducing redundant storage and computational costs.

Original authors: Wenzhi Li, Dong Nie, Rui Lan, Tongtong Lyu, Peiyao Wang, Lingzi Hong, Weihang Pan, Boyuan Pan, Yao Hu

Published 2026-08-25

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenzhi Li, Dong Nie, Rui Lan, Tongtong Lyu, Peiyao Wang, Lingzi Hong, Weihang Pan, Boyuan Pan, Yao Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, large language models are like vast libraries of human knowledge, but they have a significant limitation: they cannot easily remember new things that happen after they are built. When these models are used as assistants in dynamic environments—such as long conversations, research projects, or personal planning—they encounter a constant stream of new information. Some of this information is already known to the model, some is entirely new, and some contradicts what the model previously believed. Traditional systems handle this by simply saving every new fact into an external database, much like a notebook that never gets thrown away. Over time, this notebook becomes so full of redundant and outdated notes that finding the right information becomes slow, noisy, and inefficient. The core challenge, therefore, is not just about storing information, but about deciding what to keep, what to discard, and when to permanently learn something so it no longer needs to be looked up.

A team of researchers has proposed a new way to manage this flow of knowledge, inspired by how the human brain handles memory. They suggest that instead of treating memory as a single, growing pile of data, it should be viewed as a lifecycle with two distinct stages: a fast, temporary holding area and a slow, permanent learning phase. This approach, called Dual-Layer Agentic Memory, mimics the biological relationship between the hippocampus, which quickly records new experiences, and the neocortex, which slowly integrates stable knowledge over time. In their system, the "fast" layer acts as a temporary buffer for new or conflicting facts, while the "slow" layer involves the model actually learning these facts so they become part of its permanent understanding. The goal is to prevent the external notebook from becoming bloated while ensuring the model becomes smarter and more self-reliant over time.

The researchers built a system that makes decisions about incoming information the moment it arrives, rather than waiting until a question is asked. When a new fact enters the system, it is immediately evaluated to determine its fate. The system classifies the information into three categories: facts the model already knows perfectly, facts the model does not know at all, and facts the model knows incorrectly or has outdated information about. Instead of saving everything, the system uses a smart filter to decide what is worth storing. This filter works like a two-tiered screening process. A smaller, faster, and less expensive version of the model checks the incoming information first. If the fact is clearly something the model already knows or clearly something it should ignore, the small model makes the decision instantly. Only when the fact is ambiguous or difficult to judge does the system pass it to a larger, more powerful model for a final verdict. This "small-to-large" cascade ensures that the system does not waste computing power on easy decisions, while still maintaining high accuracy for complex ones.

Once the system decides to keep a piece of information, it stores it in the external memory. However, this is not the end of the process. The researchers introduced a second phase called "consolidation," which happens periodically. During this phase, the system reviews the high-value facts it has stored and uses them to retrain the model. This is similar to how a student might review notes before an exam to move information from short-term to long-term memory. By training the model on these selected facts, the model learns them permanently. As a result, the model no longer needs to look up these facts in the external notebook; it can answer them directly from its own internal knowledge. This creates a feedback loop: as the model learns more, it needs to store less, and the external memory remains compact and efficient.

The results of this approach were tested in a simulated environment where the agent had to manage a continuous stream of new facts and answer questions over time. The researchers found that their system could prune away up to 68% of the redundant information that would normally be saved in a traditional system. Despite storing far less data, the system maintained over 98% of the accuracy that would be achieved if it had saved every single piece of information. The use of the two-tiered filter also significantly reduced the computing cost, as the larger, more expensive model was only consulted for about 40% of the incoming facts. Furthermore, the consolidation phase proved effective at internalizing knowledge. After the model was retrained on the stored facts, it could answer questions without needing to search the external memory, and the system automatically stopped trying to save those same facts again.

However, the study also revealed a critical reason why the dual-layer design is necessary. When the model learns new information through this retraining process, it occasionally disrupts its existing knowledge, causing it to forget or confuse facts it previously knew well. This phenomenon, known as catastrophic forgetting, means that the model cannot simply overwrite its old memories with new ones without risk. The external memory layer serves as a safety net in this scenario. If the model makes a mistake after learning something new, the external memory can still provide the correct information to fix the error. This dynamic balance allows the agent to continuously learn and adapt without losing its ability to reason correctly. The researchers demonstrated that by combining fast, selective storage with slow, selective learning, they created a memory system that is both efficient and robust, capable of handling the evolving nature of real-world information without becoming overwhelmed.

This work suggests a shift in how we think about artificial intelligence memory. Rather than viewing memory as a static archive that grows indefinitely, it is more effective to view it as a dynamic process of selection and integration. The system does not just store data; it manages the lifecycle of knowledge, deciding what is temporary and what is permanent. By mimicking the biological principles of fast recording and slow consolidation, the researchers have created a framework that keeps the agent's memory lean and its reasoning sharp. The findings indicate that for AI agents to operate effectively in the long term, they must be able to forget the trivial, learn the important, and constantly update their understanding of the world without losing their way. This approach offers a promising path forward for building assistants that can truly learn and grow alongside their users.

Technical Summary: Dual-Layer Agentic Memory with Fast Write Routing and Slow Consolidation

1. Problem Statement

Large Language Model (LLM) agents operating in dynamic, long-horizon environments face a fundamental challenge: knowledge evolves continuously, yet existing memory systems treat external memory as a monotonically growing repository. Current approaches typically rely on Retrieval-Augmented Generation (RAG) or memory logs where observations are unconditionally appended. This design leads to two critical issues:

Retrieval Degradation: As external memory grows, retrieval becomes noisier, more computationally expensive, and latency-sensitive.
Static Storage Assumption: Existing methods implicitly assume a fixed storage substrate, failing to manage the full knowledge lifecycle. They do not proactively decide what to externalize, update, or internalize at the moment of entry, often leading to "memory bloat" with redundant or outdated facts.

The authors argue that the core challenge is not merely retrieval, but managing the knowledge lifecycle: deciding what to externalize, update, or ultimately internalize into parametric memory.

2. Methodology: Dual-Layer Agentic Memory

Inspired by Complementary Learning Systems (CLS) theory in neuroscience—which posits a fast-learning hippocampus for novel experiences and a slow-learning neocortex for stable consolidation—the authors propose a framework that shifts memory management to the write phase.

The framework consists of two layers:

Carrier Layer: Maintains Parametric Memory ( $\Theta$ , stable, compressed) and External Memory ( $E$ , dynamic, editable).
Decision Layer: Separates Fast Write Routing (admission control) from Slow Write-Back (consolidation).

2.1 Operational Memory Taxonomy

Incoming knowledge is categorized based on model behavior (zero-shot vs. memory-supported) into three classes:

Non-write: The model answers correctly from parametric memory alone.
Write-new: The model refuses or fails zero-shot but succeeds with memory (fact is missing).
Write-update: The model gives a confident but incorrect answer zero-shot but succeeds with memory (stale internal knowledge).

2.2 Fast Write Routing (Small-to-Large Cascade)

To avoid the computational cost of using a large model for every admission decision, the system employs a cost-aware small-to-large cascade:

Mechanism: A lightweight router ( $M_{small}$ , e.g., 1.7B model) performs front-end screening. It predicts whether a fact should be written or discarded.
Escalation Gate: A gating module estimates the value of escalation. If the small router is uncertain, the item is escalated to a stronger router ( $M_{large}$ , e.g., 8B model).
Objective: The system optimizes a trade-off between storage penalty ( $\lambda_s$ ) and escalation penalty ( $\lambda_e$ ). A fact is written only if the marginal performance gain outweighs the combined costs.
Features: Decision heads utilize semantic embeddings (mean pooling of hidden states) and epistemic uncertainty features (token-level negative log-likelihood curves).

2.3 Slow Consolidation (Write-Back)

This phase corresponds to the "sleep" phase in CLS theory. Selected external memories (specifically write-new and write-update facts) are periodically internalized into parametric memory via Supervised Fine-Tuning (SFT).

Process: Facts are converted into question-answer pairs and sentence-level targets for SFT.
Validation: A fact is evicted from external memory only if the updated model ( $\Theta'$ ) answers its probes correctly without retrieval.
Adaptive Shift: As the model internalizes knowledge, its parametric boundaries shift. The router naturally adapts, recognizing that previously "write-worthy" facts are now "non-write," thereby reducing future reliance on external memory.

3. Key Contributions

Unified Lifecycle Formulation: The paper formulates agent memory as a dual-layer system unifying external memory, parametric memory, fast write routing, and slow consolidation, moving beyond static storage paradigms.
Operational Taxonomy & Cost-Aware Routing: It introduces a behavior-based taxonomy (non-write, write-new, write-update) and a small-to-large routing cascade that frames memory admission as an accuracy-efficiency optimization problem.
Periodic Write-Back Mechanism: A consolidation mechanism that enables knowledge internalization and adaptive epistemic shift, progressively reducing external memory reliance.
Online Streaming Benchmark: The authors construct a benchmark that decouples write-time admission from read-time retrieval, isolating the admission bottleneck to evaluate selective externalization and consolidation.

4. Experimental Results

Experiments were conducted on an online streaming benchmark using a qwen3-1.7B/8B cascade.

Dual Efficiency: The cascaded router prunes up to 68% of redundant external memory while escalating fewer than 50% of inputs to the large model.
Performance Retention: Despite significant pruning, the system retains over 98% of the downstream QA Exact Match (EM) achieved by an exhaustive retention baseline (Full Store).
Cost Reduction: The cascade reduces expected routing compute costs by roughly 30–39% compared to using a large model for all decisions.
Epistemic Shift: With periodic SFT (Write Router $_{SFT}$ ), the system achieves a 32.08% storage ratio while maintaining 90.71% QA EM (retaining >98.2% of the Full Store upper bound).
Parametric Interference: Analysis of the write-back phase revealed that while SFT internalizes ~10,683 facts, it also causes parametric interference, degrading 1,752 previously stable facts. This empirically validates the necessity of the external memory layer as a buffer to correct inevitable epistemic regressions caused by periodic updates.

5. Significance and Claims

The paper claims to present a unified paradigm for agent memory: selective externalization followed by selective internalization.

Paradigm Shift: It challenges the view of external memory as a permanent dump, reframing it as a temporary, selective store for knowledge missing from or conflicting with parametric memory.
Closed Loop: The framework closes the knowledge lifecycle loop. The router senses the model's evolving epistemic boundaries; as facts are internalized, the router adaptively suppresses redundant writes, effectively managing the trade-off between storage bloat and retrieval accuracy.
Necessity of Dual-Layer: The authors conclude that while continuous parametric updates internalize vast knowledge, they inherently disrupt stable facts. Therefore, a dual-layer architecture with a fast-routing external memory layer is essential to dynamically correct these errors, acting as a necessary buffer against catastrophic forgetting in lifelong learning scenarios.

The paper acknowledges limitations, including the latency and compute cost of offline SFT write-back and the lack of a principled mechanism to resolve temporally conflicting updates during consolidation, suggesting future work in continuous, parameter-efficient adaptation.