💬 NLP

Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs

This paper introduces Neuromem, a scalable testbed that benchmarks external memory modules for LLMs under realistic, interleaved streaming conditions by decomposing the memory lifecycle into five key dimensions, revealing that while data structures primarily dictate performance quality, aggressive compression and generative integration offer limited accuracy gains despite shifting costs between insertion and retrieval.

Original authors: Ruicheng Zhang, Xinyi Li, Tianyi Xu, Shuhao Zhang, Xiaofei Liao, Hai Jin

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ruicheng Zhang, Xinyi Li, Tianyi Xu, Shuhao Zhang, Xiaofei Liao, Hai Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but forgetful assistant to help you run a complex business. This assistant (the AI) is incredibly smart but has a very short attention span; they can only hold about 10 minutes of conversation in their head at once. If you talk to them for an hour, they forget what you said at the beginning.

To fix this, you give them a notebook (External Memory). Every time you tell them something new, they write it down. When you ask a question later, they flip through the notebook to find the answer.

For a long time, researchers tested these notebooks by writing the whole book first, closing it, and then asking questions. But in real life, life doesn't work like that. You are constantly talking, the notebook is constantly getting filled, and you are asking questions while the notebook is being written. The notebook is streaming.

This paper, Neuromem, is like a giant, high-tech testing lab that simulates this messy, real-life scenario. Instead of just asking "Is the assistant smart?", they break the assistant's notebook system down into five specific parts to see exactly what works and what breaks.

Here is the breakdown using simple analogies:

The Five Parts of the Notebook System

The authors realized that a memory system isn't just one thing. It's a factory line with five stations:

The Filing Cabinet (Data Structure): How do you organize the notes?
- Option A: A simple stack of papers (FIFO Queue). You just add to the top.
- Option B: A complex library with a card catalog and cross-references (Knowledge Graph).
- The Finding: The way you organize the cabinet sets the ceiling for how smart the assistant can be. A messy pile of papers will never be as good as a well-indexed library, no matter how hard you try to fix the other parts.
The Translator (Normalization): Before writing a note, do you rewrite it?
- Option A: Write exactly what was said (Raw Text).
- Option B: Summarize it into a bullet point (Enrichment).
- Option C: Force it into a rigid "Subject-Verb-Object" format (Rewriting).
- The Finding: Don't over-edit. Trying to summarize or force facts into rigid formats destroys the nuance. It's like trying to describe a movie by only writing down the names of the actors; you lose the plot. Keeping the raw text is faster and more accurate.
The Janitor (Consolidation): How do you keep the notebook from getting too full?
- Option A: Let the AI read the notes and decide what to delete or merge (Generative).
- Option B: Use simple rules, like "delete the oldest note" or "delete notes no one has looked at in a week" (Heuristics).
- The Finding: Simple rules win. Asking the AI to "think" about what to delete takes forever and doesn't actually make the assistant smarter. A simple "delete the oldest" rule is lightning fast and just as effective for keeping the system running.
The Search Query (Query Formulation): How do you ask the assistant to find a note?
- Option A: Just ask the question directly.
- Option B: Ask the AI to break the question down into smaller parts or rewrite it first (Generative).
- The Finding: Don't overthink the search. Asking the AI to rewrite your question before searching is like asking a librarian to write a poem about your question before they look for the book. It takes too long and rarely helps you find the answer faster.
The Reader (Context Integration): How does the assistant read the notes to answer you?
- Option A: Just read the top 3 notes found.
- Option B: Ask the AI to combine the notes into a perfect summary before answering (Generative).
- The Finding: The "Latency Tax." This is the paper's biggest discovery. Using the AI to "polish" the notes before answering costs a massive amount of time (seconds) for almost zero improvement in the final answer. It's a "tax" you pay for no real benefit.

The Big Takeaways (The "So What?")

The "Streaming" Problem is Real: Most tests are fake because they assume the notebook is finished before you start asking questions. In reality, the notebook is always changing. When you test it this way, performance drops as the notebook gets bigger, especially for questions about time (e.g., "What did I say yesterday?").
Simplicity is Speed: The most efficient systems don't use the AI to manage the memory. They use simple, computer-science rules (like "delete the oldest") to manage the data, and save the "smart AI" part only for the final answer.
The "Latency Tax": Every time you try to make the system "smarter" by having the AI summarize, rewrite, or merge notes during the process, you pay a huge price in speed. It's like hiring a chef to chop vegetables, but then hiring a second chef to chop the chopped vegetables again. It takes twice as long and the salad tastes the same.

The Conclusion

If you want to build a smart AI assistant that remembers your long conversations, don't over-engineer the memory.

Use a good filing system (Data Structure).
Keep the notes raw and unedited (Normalization).
Use simple, fast rules to delete old notes (Consolidation).
Don't ask the AI to rewrite your questions or summarize the notes before answering.

The paper argues that the future of long-term memory for AI isn't about making the AI work harder on the memory; it's about making the memory system itself simpler, faster, and more like a well-organized library rather than a chaotic brainstorming session.

1. Problem Statement

Current evaluations of External Memory Modules for Large Language Models (LLMs) suffer from three critical limitations:

Static Assumptions: Most benchmarks assume a "build-then-query" paradigm where memory is constructed offline and queried at a fixed state. This fails to reflect real-world deployments where memory is streaming: new facts arrive continuously, insertions interleave with retrievals, and the memory state evolves during service.
Coarse-Grained Analysis: Existing benchmarks often treat memory systems as "black boxes," ranking entire systems without isolating the impact of specific design decisions (e.g., data structures vs. normalization strategies).
Incomplete Efficiency Metrics: Evaluations frequently report average latency or ignore the distinction between memory maintenance costs and model inference costs, obscuring where computational overheads are incurred.

The paper argues that to optimize External Memory Modules, evaluation must be streaming-aware (handling interleaved operations) and lifecycle-aware (decomposing the system into granular stages).

2. Methodology: The Neuromem Framework

The authors introduce Neuromem, an open-source, scalable testbed designed to benchmark External Memory Modules under a realistic interleaved insertion-and-retrieval protocol.

A. Granular Decomposition (The 5 Dimensions)

Neuromem decomposes the memory lifecycle into five orthogonal design dimensions to enable controlled ablation studies:

D1: Memory Data Structure: The storage substrate (e.g., Partitional/Flat Vector Stores vs. Hierarchical/Knowledge Graphs).
D2: Normalization Strategy: How raw context is preprocessed before insertion (e.g., Raw Text, Summarization, or Triplet Extraction).
D3: Consolidation Policy: Maintenance logic for state evolution (e.g., Conflict Resolution via LLM, Decay Eviction, or Heat-based Migration).
D4: Query Formulation Strategy: How user queries are transformed into retrieval signals (e.g., Direct Embedding, Keyword Extraction, or Query Decomposition).
D5: Context Integration Mechanism: How retrieved evidence is synthesized for generation (e.g., Reranking, Fusion, or Simple Concatenation).

B. Experimental Protocol

Interleaved Streaming: The system processes a strictly ordered time-series stream of requests ( $R$ ). For every INSERT, the memory state $M$ is updated; for every RETRIEVE, the current state is queried. This prevents future leakage and simulates real-time causality.
Datasets: The framework evaluates variants across three representative benchmarks: LOCOMO (long-horizon dialogue), LONGMEMEVAL (multi-session recall), and MEMORYAGENTBENCH (dynamic fact updates).
Metrics:
- Accuracy: Token-level F1 score.
- Efficiency: Granular Insertion Latency and Retrieval Latency (decoupled from model inference).

C. System Instantiations

The framework reproduces 12+ representative memory systems (e.g., MemGPT, HippoRAG, MemoryOS, TIM) by mapping their components to the D1–D5 taxonomy, allowing the authors to swap specific modules (e.g., replacing a Graph structure with a Vector Store while keeping the same normalization strategy) to isolate variables.

3. Key Contributions

Neuromem Testbed: A unified, open-source platform for evaluating External Memory Modules under streaming protocols, enabling precise attribution of performance to specific architectural choices.
Taxonomy of Memory Lifecycle: A formal decomposition of memory architectures into five design dimensions, moving beyond monolithic system comparisons.
Empirical Guidelines: Extensive empirical study revealing that storage structure dictates the accuracy ceiling, while generative optimizations often act as "latency traps" with negligible accuracy gains.

4. Key Results & Findings

The study yields seven core empirical findings (F1–F7) regarding the trade-offs between accuracy and latency:

F1: Temporal Entropy is Inherent: Performance universally degrades as interaction history accumulates. Even computationally expensive LLM-based consolidation policies fail to mitigate this decay, exhibiting similar degradation rates (~22%) to unmaintained baselines.
F2: Storage Architecture Dictates Bounds: The data structure (D1) sets the hard ceiling for reasoning quality. Hybrid Partitional structures (e.g., Inverted Index + Vector Store) anchor the efficiency frontier, outperforming pure Hierarchical graphs in reasoning benchmarks while avoiding the overhead of graph maintenance.
F3: Semantic Compression is Lossy: Aggressive normalization (D2) like converting dialogue to rigid triplets ("Rewrite") causes functional collapse, reducing F1 scores by >50% and increasing insertion latency by 10x. Preserving raw text texture is superior.
F4: Generative Optimization Incurs a Latency Tax: Strategies like query decomposition or multi-query expansion (D4) inflate processing times by an order of magnitude (often >1s) with negligible or negative accuracy gains.
F5: Heuristics Define the Efficiency Frontier: In streaming regimes, deterministic heuristics (e.g., heat-based migration, decay eviction) outperform generative maintenance (D3). They offer superior accuracy-latency trade-offs and sub-millisecond overheads.
F6: Query Formulation is a Latency Trap: Generative query enhancement (e.g., keyword extraction via LLM) adds significant latency without proportional accuracy improvements compared to direct retrieval.
F7: Context Integration is Costly: Multi-query fusion (D5) acts as a "latency tax." While stronger models (Llama-3-8B) can extract marginal gains, the latency cost (6.5x increase) is prohibitive for real-time systems.

Scalability Note: The paper highlights that iterative graph maintenance (e.g., HippoRAG) faces a quadratic latency bottleneck ( $O(N^2)$ ) as memory grows, rendering it unsuitable for strict real-time streaming without asynchronous updates.

5. Significance and Impact

Paradigm Shift: The paper challenges the prevailing trend of using heavy generative models for every step of memory management. It advocates for lightweight, deterministic heuristics for maintenance and integration, reserving LLMs for generation only.
Green AI: By identifying the "latency tax" of generative fusion and aggressive compression, the work provides a pathway to more energy-efficient and scalable AI agents.
Privacy & Compliance: Heuristic consolidation policies (like decay eviction) offer deterministic control necessary for data erasure mandates, unlike non-deterministic generative maintenance.
Future Directions: The authors suggest that future research should focus on optimizing the insert-maintain-retrieve-integrate loop using structural heuristics rather than relying on increasingly complex generative pipelines, and formalize the distinction between short-term working memory and long-horizon storage.

In summary, Neuromem demonstrates that for streaming LLM applications, simplicity and structural efficiency often outperform complex generative interventions, providing a clear roadmap for building scalable, real-time memory systems.