MegaMem: A Retrieval Solution for Ultra-Large Context Windows
MegaMem is a source-resolved dual-view retrieval system that enables accurate generation over ultra-large persistent memories (up to one billion tokens) by decoupling semantic search from bounded evidence retrieval, achieving significant performance gains on the EnterpriseRAG-Bench.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Modern artificial intelligence has become remarkably good at answering questions, but it faces a fundamental limitation: it can only hold a certain amount of information in its active "mind" at one time. Imagine trying to read a single book while simultaneously keeping the entire contents of a massive library in your head; the more books you try to remember, the harder it becomes to focus on the one you are actually reading. For years, researchers have tried to solve this by building systems that store vast amounts of data outside the computer's immediate attention, retrieving only the specific pages needed when a question is asked. This approach works well for small collections, but it begins to falter when the library grows to contain hundreds of millions of pages, such as the complete code for a software company or years of corporate records. The challenge is not just finding the right page in such a huge pile, but doing so without overwhelming the computer with too much text, which would slow it down or confuse its answer.
A team of researchers has developed a new system called MegaMem to solve this specific problem. Their work addresses a critical bottleneck in how artificial intelligence handles massive, persistent memories. Instead of trying to force the computer to read through millions of documents every time it is asked a question, the team created a two-step process that separates the act of searching from the act of answering. They built a system that first looks for answers using highly condensed summaries of the information, and only once a promising lead is found does it retrieve the full, detailed text. This allows the system to search through a library containing hundreds of millions of words while only feeding the computer a small, manageable amount of text to generate an answer.
The researchers tested their system on a massive dataset of over 500,000 real-world enterprise documents, totaling roughly 650 million words. This collection included everything from technical manuals and legal contracts to meeting notes and product specifications. In their experiments, they compared MegaMem against standard methods that try to search the raw documents directly. The results were striking. When the system was asked to find information within this enormous library, it successfully improved the overall quality of its answers from about 68 percent to over 82 percent. More importantly, it maintained a high level of accuracy even as the size of the library grew from 20 million words to 250 million words. This demonstrated that the system could scale up to handle a library nearly the size of a small nation's entire written output without losing its ability to find the correct facts.
The secret to this success lies in how the system organizes and retrieves information. When the library is first built, the computer reads every document and creates two different versions of the data. One version consists of the original, detailed text, kept exactly as it was written. The second version is a set of distilled, compact notes that capture the main ideas, facts, and procedures from those documents. When a user asks a question, the system first searches these compact notes. Because the notes are short and focused, the computer can scan through the entire library of millions of words in a fraction of a second. If the search finds a relevant note, the system then uses a unique identifier to locate the exact original document that the note came from. It retrieves only that specific piece of detailed text to help the computer formulate its final answer. This ensures that the computer has the precise, unaltered evidence it needs without being distracted by irrelevant information.
The researchers also found that this method is far more efficient than previous approaches. By using the condensed notes to guide the search, the system reduced the amount of text it needed to process for each answer by nearly 70 percent, yet it still produced answers that were almost as accurate as if it had read the full documents. This efficiency is crucial because it means the system can remain fast and responsive even as the memory it manages grows larger. The study also revealed that the main difficulty in handling such large libraries is not the computer's ability to understand the text once it finds it, but rather the difficulty of finding the right text in the first place. As the library grew, the system's performance dipped slightly, but this was due to the challenge of locating the correct evidence among more distractions, not because the computer failed to understand the evidence once it was found.
To ensure the system was trustworthy, the researchers added a final step where the computer reviews its own answer and identifies exactly which documents supported its claims. This process filtered out any sources that were retrieved but not actually used, resulting in a cleaner and more accurate list of references. The team tested their system on various types of questions, including those requiring complex reasoning across multiple documents and those involving conflicting information. In every case, the two-step approach of searching summaries first and then retrieving details proved superior to methods that tried to search the raw text directly. The work suggests that for artificial intelligence to truly function as a long-term memory for organizations, it must separate the scale of what it can search from the size of what it can read at one time. By doing so, MegaMem offers a practical path toward building systems that can remember and reason over the vast, complex histories of modern enterprises without getting lost in the noise.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.