🤖 AI

MEMONDEMAND: A Memory Management System for Large-Scale Enterprise Data

MEMONDEMAND is a scalable memory management system for large-scale enterprise data that employs a dynamic multi-level hierarchy, dual memory separation, and on-demand promotion to outperform existing retrieval methods across various data scales and domains.

Original authors: Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world, large organizations generate a staggering amount of information every day. Policies, contracts, emails, code, and records pile up into massive digital repositories that can contain hundreds of millions of documents. When a human needs to find a specific fact within this ocean of data, they rely on search tools. Today, artificial intelligence systems are increasingly used to act as these search tools, reading the documents and answering questions. However, these AI systems have a significant limitation: they can only hold a small amount of information in their active memory at one time, much like a person trying to remember a conversation while only keeping the last few sentences in mind. If the answer lies deep within a massive collection of files, the AI often misses it or gets confused by irrelevant details. Researchers have tried to solve this by giving the AI a permanent memory bank, but existing methods struggle to organize such vast amounts of data efficiently without overwhelming the system or losing the precise details needed to prove an answer is correct.

A team of researchers has introduced a new system called MEMONDEMAND to solve this problem. The core idea is to stop trying to load every single document into the AI's active memory at once. Instead, the system builds a flexible, multi-layered structure for the data, similar to how a library organizes books into broad categories, then sections, and finally individual shelves. At the very top of this structure, the system keeps a compressed, high-level summary of the information. This summary is lightweight and allows the AI to quickly scan the entire collection to find which general area might contain the answer. As the system narrows down the search, it moves deeper into the layers, eventually reaching the original, detailed documents only when absolutely necessary. This approach ensures the AI spends its limited memory on the most relevant facts rather than wasting space on irrelevant text.

The researchers tested this system on a massive dataset containing 618 million tokens of text, which represents a collection of roughly 500,000 enterprise documents. They compared their new system against the strongest existing methods available. The results showed that MEMONDEMAND was significantly more accurate. At the smallest scale of their test, with 10 million tokens, the new system improved the quality of answers by 12.23 percent compared to the previous best method. Even at the largest scale, with the full 618 million tokens, it still outperformed the competition by 4.66 percent. Crucially, the system achieved this while using far fewer resources. By separating the "search" memory from the "answer" memory, the system reduced the amount of data it needed to process for each question by more than 70 percent compared to methods that tried to load full details immediately.

A key feature of this system is its ability to adapt over time. Just as a human might remember a frequently used file cabinet drawer better than one they rarely open, the system learns which parts of the data are most useful. It tracks which documents are accessed often and keeps them readily available, while pushing less useful or outdated information into the background. This happens automatically without the system needing to rebuild its entire structure. The researchers found that this adaptive behavior allowed the system to prepare for new questions in just 1.46 seconds, whereas a traditional approach that tried to prepare everything in advance would have taken nearly ten minutes. This speed difference highlights the efficiency of only preparing the memory that is actually needed for the current task.

The study also demonstrated that this approach works well across different types of data, not just the specific corporate documents used in the main test. When the researchers applied the system to financial documents, complex multi-step reasoning questions, and fact-retrieval tasks, it maintained high performance. This suggests that the method is robust enough to handle the messy, changing nature of real-world information. The system does not just find answers; it also ensures that every answer comes with a clear citation to the exact source document, preserving the integrity of the information. By managing memory dynamically, the system allows artificial intelligence to work effectively with libraries of data that are far larger than the AI's own internal capacity, bridging the gap between massive enterprise records and the limited attention span of current AI models.

Technical Summary: MEMONDEMAND

Problem Formulation

Enterprise repositories present a unique retrieval challenge characterized by massive scale (hundreds of millions to billions of items), heterogeneity (policies, contracts, emails, code, tickets), and continuous evolution. Existing Retrieval-Augmented Generation (RAG) and memory systems face three primary limitations when applied to this domain:

Rigid Hierarchy: Fixed taxonomies or predefined hierarchies fail to accommodate the diverse structures of different enterprise collections and tenants.
Inefficient Memory Trade-offs: Systems often force a choice between compressed memory (efficient for routing but prone to omitting critical details) and detailed memory (faithful to sources but expensive to search and load).
Static State Management: Preparing all memory representations upfront wastes resources on unused data, while failing to adapt to changing workloads leads to stale or irrelevant active states.

The core problem is to design a memory system that supports accurate, source-faithful retrieval and generation across ultra-large scales while maintaining efficiency and adapting to dynamic query patterns without exceeding context budgets.

Methodology: MEMONDEMAND

MEMONDEMAND (On-Demand Memory) addresses these challenges through a unified management design comprising three coordinated mechanisms:

1. Dynamic Multi-Level Hierarchy

Instead of a fixed taxonomy, MEMONDEMAND constructs a hierarchy specific to each collection.

Construction: Starting from Level 0 (L0) source records, the system clusters distilled representations and proposes higher abstraction levels (L1, L2, etc.). An LLM controller evaluates these proposals based on group coherence, residual noise, and budget constraints, deciding to CREATE, RECLUSTER, or STOP.
Navigation: At query time, retrieval begins at the highest available level and descends hierarchically (L $\ell$ $\to$ L0). This coarse-to-fine approach narrows the search space efficiently while preserving direct access to L0 source nodes if higher-level abstractions lack necessary details.

2. Dual Memory at Every Level

To balance retrieval cost with evidence fidelity, every node in the hierarchy maintains two distinct representations:

Distilled Memory: A compact representation containing topics, entities, identifiers, and decision-relevant facts. This is used for efficient routing and ranking throughout the hierarchy traversal.
Detailed Memory: The full source content (at L0) or complete abstraction (at higher levels). This is not loaded during routing.
Selection: Only after the hierarchy traversal identifies the most relevant L0 candidates does the system load the detailed memory for the top-ranked sources that fit within the answer-input token budget ( $B$ ). This ensures the generation model receives rich evidence without incurring the token cost of loading full content for every candidate.

3. On-Demand Memory Promotion

MEMONDEMAND adapts to workload changes by managing node priority dynamically under a bounded active-state budget ( $K$ ).

Promotion: When a node is accessed, a promotion controller calculates a score based on query relevance and usage history. If the score exceeds a threshold ( $\theta$ ), the node is promoted, increasing its retrieval priority.
Decay and Demotion: Retention scores decay over time ( $\rho_t(v)$ ). If the active state exceeds the budget $K$ , nodes with expired retention or low scores are demoted.
Efficiency: This mechanism avoids "eager" preparation of all parent-level representations. Representations are only constructed or prioritized when query evidence indicates they are useful, significantly reducing upfront computation.

Key Contributions

System Design: Introduction of MEMONDEMAND, a memory management system that jointly handles hierarchical organization, retrieval representation, evidence loading, and cross-query state updates for large-scale enterprise data.
Three Coordinated Mechanisms:
- A dynamic multi-level hierarchy that infers structure and depth per collection.
- Dual memory at every level, separating efficient routing from source-faithful answering.
- On-demand promotion that adapts node priority to workload changes within a bounded budget.
Empirical Validation: Demonstration of successful operation at ultra-large scales (up to 618M tokens) with significant performance gains over existing baselines.

Results

The system was evaluated on EnterpriseRAG-Bench (500k documents, up to 618M tokens) and three external benchmarks (FinanceBench, HotpotQA, FRAMES).

Scaling Performance: MEMONDEMAND outperformed the strongest published baseline (LB#1) across all scales.
- At 10M tokens, it achieved a 12.23% gain in the Combined score.
- At the full 618M-token collection, it maintained a 4.66% gain (72.88% vs. 68.22%).
- It achieved 80.0% Correctness and 74.28% Completeness at the largest scale, despite a drop in Document Recall (48.77%), indicating the system prioritizes high-quality evidence over sheer volume.
Efficiency:
- Token Reduction: Selective detailed-memory loading reduced answer-input tokens by 70.2% (at 20M scale) compared to loading detailed memory for all candidates, while maintaining high quality.
- Preparation Time: On-demand promotion reduced the wall time for preparing parent representations from an estimated 593.5 seconds (eager preparation) to 1.46 seconds, a 407 $\times$ reduction.
Generalization: The system maintained strong performance on FinanceBench (80.0% DocRcl), HotpotQA (78.4% Correct), and FRAMES (88.76% DocRcl), proving its applicability beyond the primary enterprise benchmark.

Significance and Claims

The paper claims that MEMONDEMAND establishes a practical, accurate, and scalable solution for extending Large Language Models (LLMs) with ultra-large memory beyond model context window limits.

Scalability: It proves that managed memory systems can operate effectively at the scale of hundreds of millions of tokens, a regime where standard retrieval or fixed-context approaches fail.
Source Faithfulness: By separating routing (distilled) from evidence (detailed) and enforcing strict citation rules, the system ensures that compressed records guide access without replacing authoritative sources.
Adaptability: The on-demand promotion mechanism demonstrates that enterprise memory can adapt to changing data and workloads without incurring the prohibitive costs of full pre-computation.

The authors note limitations, including that results are based on single runs (variance not studied) and that Document Recall decreases at the largest scales, suggesting that strong source recovery does not always guarantee perfect answer correctness. Future work is suggested to explore broader domains and longer query streams.