💬 NLP

MeMo: Memory as a Model

The paper introduces MeMo, a modular framework that encodes new knowledge into a dedicated memory model to enable plug-and-play integration with frozen LLMs, offering robust retrieval of complex cross-document information without catastrophic forgetting or dependency on model weights.

Original authors: Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F. Chen, Bryan Kian Hsiang Low, Daniela Rus, Armando Solar-Lezama

Published 2026-05-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryan Wei Heng Quek, Sanghyuk Lee, Alfred Wei Lun Leong, Arun Verma, Alok Prakash, Nancy F. Chen, Bryan Kian Hsiang Low, Daniela Rus, Armando Solar-Lezama

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Frozen Brain"

Imagine a brilliant, all-knowing librarian (a Large Language Model or LLM) who has read every book in the world up to a certain date. This librarian is incredibly smart and can answer almost any question. However, there is a catch: their brain is frozen. Once they finish their training, they cannot learn anything new. If you ask them about a news story from yesterday or a specific medical breakthrough from last week, they are clueless.

To fix this, we usually try to do one of three things, but they all have flaws:

The "Stack of Papers" method (Retrieval): We hand the librarian a stack of papers and say, "Read these and answer." The problem? The librarian can only hold so many papers in their hands at once. If the answer requires connecting dots across 50 different documents, they get overwhelmed and miss the connection. Also, if the stack contains a lot of junk (noise), they get confused.
The "Brain Surgery" method (Retraining): We try to retrain the librarian's brain with new information. The problem? This is incredibly expensive, takes a long time, and often causes the librarian to forget everything they knew before (a phenomenon called "catastrophic forgetting").
The "Secret Note" method (Latent Memory): We try to compress new info into a tiny, secret code inside the librarian's brain. The problem? This code is written in a language only that specific librarian understands. You can't give this code to a different librarian.

The Solution: MEMO (Memory as a Model)

The authors propose a new system called MEMO. Instead of trying to fix the librarian's frozen brain or hand them a messy stack of papers, they build a specialized, external assistant just for them.

Think of it like this:

The Executive Model (The Librarian): This is the main AI. It stays frozen and unchanged. Its job is to think, reason, and talk to you.
The Memory Model (The Specialist Assistant): This is a smaller, dedicated AI trained only on the new information you want the librarian to know. It acts like a super-organized, hyper-specialized encyclopedia that the librarian can consult.

How It Works: The Three-Step Conversation

When you ask a complex question, the system doesn't just "search" for an answer. It uses a structured, three-step conversation between the Librarian and the Specialist:

Grounding (The "What are we talking about?" phase):
The Librarian breaks your big, complex question into tiny, simple clues. It asks the Specialist: "Who is the main person here?" or "What year is this about?" The Specialist answers with short, precise facts.
Entity Identification (The "Pinpointing" phase):
Using those clues, the Librarian tries to narrow down exactly who or what you are asking about. It keeps asking the Specialist follow-up questions until it's 100% sure of the target (e.g., "Is it Linda the nurse, or Linda the doctor?").
Answer Synthesis (The "Putting it together" phase):
Once the target is pinned down, the Librarian asks the Specialist for the specific details needed to answer your question. The Librarian then takes those clean, verified facts and writes the final answer for you.

How They Trained the Specialist

You might wonder: "How do you teach this Specialist without just dumping raw data on it?"
The authors created a Data Synthesis Pipeline. Imagine a "Generator" (a smart AI) reading the raw documents and turning them into a Q&A training manual for the Specialist.

It extracts facts.
It merges related facts (e.g., "Linda is the nurse" + "Linda works in the ER" becomes "Linda is the ER nurse").
It checks for errors and rewrites confusing questions.
It creates "cross-document" questions that require connecting info from different pages.

This ensures the Specialist learns the relationships between facts, not just random sentences.

Why MEMO is Better (The Superpowers)

The paper claims MEMO solves the problems of the old methods:

It handles complex connections: Because the Specialist has "internalized" the relationships between facts, it can answer questions that require connecting dots across many documents, which the "Stack of Papers" method struggles with.
It ignores the noise: If you give the "Stack of Papers" method a bunch of fake news articles, it gets confused. MEMO's Specialist has already filtered and learned the truth, so it ignores the junk.
No brain surgery: The main Librarian (Executive Model) never changes. You don't need to retrain it, so it never forgets its original knowledge.
Plug-and-Play: Because the Specialist speaks a standard language (questions and answers), you can use it with any librarian, whether it's an open-source one or a paid, closed-source one (like a corporate AI). You don't need to see their internal code.
Speed: The cost to find an answer doesn't get more expensive just because your library of documents gets huge. The Specialist is a fixed size, so the search time stays constant.

The Results

The authors tested this on three difficult challenges:

BrowseComp-Plus: Deep research requiring multi-step digging.
NarrativeQA: Understanding long stories (like books or scripts).
MuSiQue: Answering questions that require combining facts from multiple Wikipedia pages.

In these tests, MEMO beat the best existing methods (like standard search engines or methods that try to retrain the AI). It was particularly good at handling "noise" (irrelevant information) and solving complex, multi-step puzzles.

One Catch: The "Merging" Trick

The paper also shows a cool trick for when you have multiple new topics to learn (e.g., new medical data, then new legal data). Instead of retraining the Specialist from scratch every time, you can train a small Specialist for the medical data, another for the legal data, and then merge their "brains" (mathematically combining their weights) into one super-Specialist. This saves a massive amount of computing power compared to retraining from scratch, though it might be slightly less accurate than a full retrain.

In summary: MEMO separates the "thinking" from the "knowing." It keeps the main AI frozen and smart, while giving it a dedicated, highly-trained assistant that knows the new stuff perfectly, allowing them to work together to answer complex questions without forgetting anything or getting confused by noise.

Technical Summary: MEMO (Memory as a Model)

Problem Statement

Large Language Models (LLMs) achieve strong performance across diverse tasks but suffer from a fundamental architectural limitation: their knowledge is frozen after pretraining. As the world evolves, these models rely on static knowledge that becomes increasingly outdated. While retraining is a natural solution, it is prohibitively expensive at modern scales and prone to catastrophic forgetting. Existing methods for integrating new knowledge fall into three categories, each with significant drawbacks:

Non-parametric methods (e.g., RAG, ICL): These retrieve information at inference time but are constrained by context window limits, struggle to synthesize complex cross-document relationships, and are highly sensitive to retrieval noise.
Parametric methods (e.g., continual pretraining, fine-tuning): These internalize knowledge into model weights but are computationally expensive, prone to catastrophic forgetting, and often infeasible for proprietary, closed-source models.
Latent memory methods: These compress knowledge into soft tokens but suffer from "representation coupling," where the memory is tightly bound to the specific model architecture that produced it, limiting transferability.

The paper posits the need for an efficient mechanism to integrate new external knowledge into LLMs without full retraining, one that captures complex relationships, is robust to noise, avoids forgetting, and supports plug-and-play integration with both open and proprietary models.

Methodology

The authors introduce MEMO (Memory as a Model), a modular framework that decouples knowledge storage from reasoning. MEMO consists of two primary components: a MEMORY model ( $M_\phi$ ) trained to encode new knowledge, and a frozen EXECUTIVE model ( $M_\theta$ ) that handles reasoning and user interaction.

1. Data Synthesis Pipeline

To train the MEMORY model, the authors propose a novel five-step data synthesis pipeline guided by a GENERATOR model (an LLM that may be the same as or smaller than the EXECUTIVE model). This pipeline transforms a target corpus $\mathcal{D}$ into a "reflection" QA dataset ( $Q_{final}$ ) designed to capture both single-document facts and cross-document relationships:

Step 1: Fact Extraction: The generator performs direct extraction (explicit facts) and indirect extraction (inferred/synthesized information) from document chunks.
Step 2: Consolidation: Related QA pairs sharing a common context (entity, time, relationship) are merged into multi-fact questions to encourage compositional representations.
Step 3: Verification and Rewriting: Pairs are evaluated for self-containment (can they be answered without the source text?). Ambiguous pairs are rewritten or discarded.
Step 4: Entity Surfacing: The generator creates QA pairs that explicitly encode entity attributes and relationships, training the model to infer identities from indirect descriptions (mitigating the "reversal curse").
Step 5: Cross-Document Synthesis: The generator identifies connections across document groups, such as "converging clues" (multiple docs identifying one entity) and "parallel properties" (shared attributes across entities), creating QA pairs that require multi-document reasoning.

Crucially, no document identifiers or watermarks are embedded, preventing the model from exploiting shortcut signals.

2. Training the MEMORY Model

The MEMORY model is initialized from a small pretrained language model (e.g., 1.5B or 14B parameters) and trained via Supervised Fine-Tuning (SFT) on the synthesized reflection QA dataset. The loss function minimizes next-token prediction on answer tokens, conditioning only on the question and preceding answer tokens. This forces the model to internalize knowledge parametrically rather than relying on copying from retrieved context.

3. Continual Integration via Model Merging

To support incremental updates without retraining on the union of all corpora, MEMO utilizes model merging. When a new corpus arrives, a new MEMORY model is trained independently. The task vectors ( $\tau_i = \phi_i - \phi_0$ ) of these models are then merged (e.g., using TIES-merging) to create a single updated MEMORY model. This approach scales linearly with the number of new corpora ( $\Theta(K)$ ) rather than quadratically ( $\Theta(K^2)$ ) as required by full retraining.

4. Inference-Time Integration

At inference, the frozen EXECUTIVE model queries the MEMORY model through a structured multi-turn protocol consisting of three stages:

Stage 1 (Grounding): The EXECUTIVE model decomposes the user query into atomic, clue-probing sub-questions. The MEMORY model answers these independently to provide contextual grounding.
Stage 2 (Entity Identification): Using grounding responses, the EXECUTIVE model iteratively issues follow-up queries to narrow down candidate entities until a single entity is identified or the budget is exhausted.
Stage 3 (Answer Seeking and Synthesis): Conditioned on the identified entity, the EXECUTIVE model queries for supporting facts and synthesizes a final answer.

This protocol treats the MEMORY model as an external knowledge oracle, requiring no access to the EXECUTIVE model's weights, gradients, or logits, enabling plug-and-play integration with any LLM.

Key Contributions

Novel Data Synthesis Pipeline: A five-step process that distills a corpus into "reflections" (compositional QA pairs) capturing explicit facts and implicit cross-document relationships, enabling robust generalization to unseen query variations.
Structured Multi-Turn Protocol: A systematic inference mechanism that decomposes complex queries into targeted sub-queries aligned with the reflection interface. This supports plug-and-play integration with arbitrary LLMs (including closed-source) and ensures retrieval cost is independent of corpus size.
Empirical Validation: Comprehensive evaluation on three benchmarks (BrowseComp-Plus, NarrativeQA, MuSiQue) demonstrating strong performance against parametric and non-parametric baselines, along with empirical validation of robustness to retrieval noise.

Experimental Results

The authors evaluated MEMO on three knowledge-intensive benchmarks using Qwen2.5-32B-Instruct and Gemini-3.0-Flash as EXECUTIVE models:

Performance: MEMO consistently outperformed baselines (BM25, NV-Embed-V2, HippoRAG2, Cartridges) on NarrativeQA and MuSiQue. On NarrativeQA, MEMO achieved 26.85% (Qwen2.5-32B) and 53.58% (Gemini-3-Flash), substantially surpassing retrieval-based methods which struggle with long-document synthesis. On MuSiQue, MEMO achieved 48.30% and 58.70% respectively.
Robustness to Noise: In ablation studies adding negative (distractor) documents, retrieval-based methods (NV-Embed-V2, HippoRAG2) suffered significant accuracy drops (up to 6.22 percentage points). In contrast, MEMO maintained stable performance, showing only a marginal decline (1.77 pp) or slight improvement, demonstrating its resilience to retrieval noise.
Scalability and Merging: Model merging experiments on NarrativeQA showed that merging two independently trained MEMORY models incurred a 33% reduction in cumulative compute cost compared to full retraining, with only a modest accuracy trade-off (11.0 pp drop with Qwen2.5-32B). The merged model still outperformed all retrieval baselines.
Plug-and-Play Capability: MEMO demonstrated that a single MEMORY model trained with a weaker GENERATOR could be seamlessly paired with a more capable EXECUTIVE model (e.g., Gemini-3-Flash), yielding significant performance gains without additional training.

Significance and Claims

The paper claims that MEMO addresses the core limitations of existing knowledge integration paradigms by:

Bypassing Context Constraints: Unlike RAG, MEMO captures complex cross-document relationships through parametric internalization rather than relying on limited context windows.
Avoiding Catastrophic Forgetting: By keeping the EXECUTIVE model frozen, MEMO avoids the degradation of previously acquired knowledge and safety alignments associated with parametric updates.
Decoupling Representation: Unlike latent memory methods, MEMO's memory is not tied to a specific model architecture, allowing for transferability across diverse LLM families.
Enabling Black-Box Integration: The framework requires no access to model weights or logits, making it applicable to proprietary, closed-source models where fine-tuning is impossible.

The authors conclude that MEMO offers a scalable pathway for knowledge integration, supporting efficient updates and flexible deployment. They acknowledge limitations regarding upfront training costs and the representational capacity of the MEMORY model relative to corpus size, viewing MEMO as a foundational step toward more flexible, updatable, and knowledge-aware AI systems.