💻 computer science

User as Engram: Internalizing Per-User Memory as Local Parametric Edits

The paper proposes "User as Engram," a novel personalization architecture that stores individual user facts as sparse, hash-keyed local parametric edits within a shared memory table while keeping reasoning skills in a global adapter, thereby achieving superior reasoning accuracy, perfect composability across users, and a drastically smaller memory footprint compared to traditional retrieval systems or per-user LoRA adapters.

Original authors: Bojie Li

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bojie Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Size-Fits-All" Memory

Imagine you have a super-smart AI assistant. You want it to remember specific things about you—like your doctor's name, your allergies, or that you went vegetarian last year.

Currently, there are two main ways AI tries to do this, and both have flaws:

The "Notebook" Method (Retrieval): The AI doesn't remember you; it just looks up a notebook file every time you ask a question. It's accurate, but it's slow and uses up a lot of "brain space" (context) to read the note.
The "Brain Surgery" Method (LoRA): The AI tries to permanently change its own brain weights to remember you. The problem is, this is like trying to write a new note on a piece of paper that is already covered in ink. To write your name, the AI accidentally smudges the ink everywhere else. This "smudging" (contamination) makes the AI worse at answering questions about things unrelated to you.

The Solution: "User as Engram"

The authors propose a new way to give AI personal memory that mimics how the human brain works. They call it "User as Engram."

To understand it, think of the human brain as having two distinct systems:

The Hippocampus (The Fast, Local Filing Cabinet): This stores specific, new memories (like "My doctor is Dr. Vasquez"). It's sparse and local.
The Neocortex (The Slow, Shared Library): This stores general skills and reasoning (like "How do I figure out which doctor to visit?"). This is shared by everyone.

User as Engram splits the AI's memory into these two exact layers:

1. The "Filing Cabinet" (Your Personal Facts)

Instead of rewriting the AI's entire brain, the system writes your specific facts into a tiny, separate "filing cabinet" inside the model.

How it works: Imagine the AI has a giant book of rules. When you say "My doctor is Dr. Vasquez," the system doesn't rewrite the whole book. It finds a specific, empty page (a specific address) in the back of the book and writes only that sentence there.
The Magic: Because it only writes to that one specific page, it doesn't smudge the rest of the book. If you ask about the weather, the AI ignores your doctor's name entirely.
The Result: You can have thousands of users, and their "filing cabinets" can all sit in the same book without overlapping. It's incredibly small (about 88 KB for 100 facts) compared to the massive "brain surgery" method (14 MB).

2. The "Shared Library" (The Reasoning Skill)

The AI still needs to know how to use those facts. If you ask, "I'm visiting my daughter in California; where should I go for a check-up?", the AI needs to reason that "California" + "Doctor" = "Find a doctor in CA."

How it works: Instead of teaching this reasoning skill to every single user individually (which causes the "smudging" problem), the AI trains one single, shared skill for everyone.
The Magic: This shared skill is like a universal translator. It knows how to take a fact from anyone's filing cabinet and use it to answer a question.

Why This is Better (The "Glass Box" Analogy)

The paper claims this method is a "glass box" because you can see exactly what is happening.

Old Way (LoRA): Like trying to fix a leak in a dam by pouring concrete everywhere. You fix the leak, but you also block the river flow for everyone else.
New Way (Engram): Like installing a specific, removable valve. When you need to fix the leak, you open that one valve. When you don't, it's closed. It doesn't affect the river flow at all.

The Results: What the Paper Actually Found

The researchers tested this against the old methods and found:

No Smudging: Writing a user's facts into the "filing cabinet" did not make the AI worse at answering questions about other topics. In fact, it was 33,000 times less damaging to the AI's general knowledge than the old "brain surgery" method.
Better Reasoning: The new method was 5.6 times better at answering indirect questions (like the California example) compared to the old method.
Scalability:
- If you have a small number of facts (under 100), the old "notebook" method (Retrieval) is still okay.
- But once you have more than 100 facts per user, the "filing cabinet" method wins. It stays fast and accurate, while the "notebook" method gets slower and confused as the list of facts grows.
Privacy: Because each user's facts are in their own unique "slot" in the filing cabinet, User A's facts can never accidentally leak into User B's answers.

The Limitations (What It Can't Do)

The paper is honest about what this method cannot do yet:

It's a "Recite" machine, not a "Think" machine: The system is great at remembering "My doctor is Dr. Vasquez." But if you ask, "Where does Dr. Vasquez work?" and that fact wasn't explicitly written down, the system can't figure it out on its own. It matches words, it doesn't chain logic across multiple facts yet.
It needs a specific base: This method requires an AI model that was specifically pre-trained to have these "filing cabinets" (called an Engram model). You can't just plug this into any standard AI model today.

Summary

User as Engram is a new way to give AI personal memory. Instead of forcing the AI to rewrite its entire brain to remember you (which breaks its other skills), it gives you a tiny, private notebook inside the AI's brain and teaches the AI a single, shared skill on how to read that notebook. This keeps the AI smart, fast, and private, without "smudging" its memory for anyone else.

Technical Summary: User as Engram

1. Problem Statement

The paper identifies a fundamental architectural flaw in current methods for personalizing Large Language Models (LLMs) with user-specific memory. Personal memory consists of two distinct components: content (specific facts about a user, e.g., "My cardiologist is Dr. Vasquez") and reasoning skill (the ability to synthesize those facts into answers).

Current approaches conflate these two:

Non-parametric methods (In-Context Learning, RAG) keep facts outside the weights. While they avoid contaminating the model, they incur high context-token costs and rely on retrieval accuracy, which degrades as the knowledge base grows.
Parametric methods (Per-user LoRA adapters) write facts directly into the model weights. However, because LoRA edits are global (affecting Q/K/V/O matrices across the entire model), they suffer from architectural contamination. Writing a user's facts as a LoRA delta bends the model's function globally, degrading performance on text unrelated to that user and often impairing the model's ability to reason indirectly over the stored facts.

The core challenge is to internalize user facts into the model weights without contaminating unrelated text or sacrificing the shared reasoning capabilities of the base model.

2. Methodology: User as Engram

The authors propose User as Engram, a layered architecture that separates content storage from reasoning skill, mirroring the brain's complementary learning systems (hippocampus for episodic traces, neocortex for shared skills).

A. The Engram Memory Table (Content Layer)

Instead of editing global weights, the method utilizes an Engram model (a sparse architecture introduced by Cheng et al., 2026) which includes a content-addressable memory table alongside standard transformer layers.

Mechanism: At specific layers, the model hashes the suffix N-gram of recent tokens into deterministic addresses in a large embedding table ( $E$ ).
Surgical Edits: A user's fact is written not as a weight delta, but as a local row override in this memory table.
- Addressing: The trigger phrase (e.g., "My cardiologist is") hashes to a specific set of rows ( $R_f$ ).
- Writing: The system writes a value vector into these specific rows. Three strategies are used:
  1. UNEMBED_P: Closed-form solution (fast, <1ms).
  2. OPT: Gradient steps per fact.
  3. Joint OPT: Optimizes all rows for a user simultaneously to prevent interference (default for >30 facts).
Locality: The edit is "glass box" transparent. It only activates when the specific trigger N-gram is present. For all other text, the memory table remains unchanged, and the residual stream is unaffected to the last bit.

B. Shared Reasoning Adapter (Skill Layer)

The reasoning skill is decoupled from the user's facts.

A single shared LoRA is trained on a held-out population of users.
This adapter learns the pattern of reasoning over facts (e.g., "Given facts X, Y, and Z, answer question Q") without memorizing specific user content.
At inference, the system loads the shared LoRA and swaps in the specific user's Engram row overrides.

C. Composition and Serving

Additive Composition: Because different users' facts hash to disjoint addresses in the memory table, their overrides are additive and lossless. Multiple users can share a single model instance without a combiner.
EngramServer: A lightweight serving system (50 lines of code) that manages per-user override maps in CPU memory (DRAM), swapping them into the GPU memory (HBM) for the forward pass and restoring them immediately after. This incurs negligible latency (0.03ms to 2ms).

3. Key Results

A. Elimination of Contamination

Unrelated Text: Writing a fact as a per-user LoRA increases the validation bits-per-byte (bpb) on unrelated text by ~1.78 (a >33,000× increase in loss compared to the baseline). In contrast, writing the same fact as an Engram row results in a change of only +0.00005 bpb, effectively zero.
Reasoning Preservation: On instruction-tuned bases, per-user LoRAs often degrade indirect reasoning (85% of users show worse performance on weak bases). The layered Engram design maintains the base's reasoning capabilities, with zero users performing worse than the untouched base.

B. Recall and Reasoning Performance

Direct Recall: The layered design (Engram + Shared LoRA) achieves near-perfect direct recall (100%), matching per-user LoRA.
Indirect Reasoning: The layered design significantly outperforms per-user LoRA on indirect questions (reasoning over facts). On a canonical seed, it achieves 44% indirect recall compared to 6% for per-user LoRA (a 5.6× to 7.4× improvement).
Comparison with Retrieval (RAG):
- On exact-trigger queries, RAG performs well but pays a context-token cost.
- On paraphrased queries, RAG performance drops significantly due to encoder limitations, while multi-trigger Engram insertion maintains high recall (96.9% vs. 75% for RAG).
- Scalability: As the knowledge base (KB) grows, RAG recall degrades due to candidate pool confusion. The layered design's accuracy remains invariant to KB size. The paper identifies a crossover point at ~100 facts: beyond this, the layered design on a smaller model outperforms RAG on a model 2.5× larger.

C. Efficiency

Storage: A user's Engram table is 88 KB for 100 facts. A per-user LoRA is ~14.2 MB. This represents a **161× reduction** in storage per user.
Latency: The override application is sub-millisecond, allowing high throughput (232 req/s on a single GPU) that scales flatly with the number of users.

4. Limitations

The paper explicitly notes the following constraints:

Multi-hop Reasoning: The Engram mechanism is surface-trigger keyed. It matches words but does not compose them. If a question requires chaining two facts ("My doctor is Patel; Patel works at Globex; where does my doctor work?"), recall collapses unless the specific chain is explicitly stored or the query matches the second fact's trigger exactly.
Within-User Density: While Joint OPT mitigates interference, recall degrades as the number of simultaneous facts per user increases (dropping to ~35% at 1,000 facts) due to forward-pass interference among co-active rows.
Pretraining Requirement: The method requires an Engram-pretrained base model. The authors trained four Mini-Engram models (178M to 1.22B parameters) to demonstrate feasibility but note that production deployment depends on the availability of such weights.

5. Significance and Claims

The paper claims that User as Engram solves the "content vs. skill" conflict in personalization by architectural separation rather than parameter sharing.

Architectural Innovation: It is the first method to achieve local parametric edits with zero context cost. It occupies a unique quadrant in the design space: local edits that do not contaminate the rest of the model.
Scalability: It enables serving millions of users on a single model instance with additive, lossless composition, avoiding the storage explosion of per-user LoRAs and the retrieval degradation of RAG.
Biological Analogy: The work validates a computational implementation of Complementary Learning Systems theory, demonstrating that separating fast, sparse episodic traces (Engram rows) from slow, distributed skills (Shared LoRA) prevents catastrophic interference and preserves reasoning capabilities.

The authors conclude that the "trouble" in personalization is not unique to adapters but is inherent to any method that forces a single set of weights to act as both memory and mind. By giving each user a small, private trace beside a shared skill, personalization stops fighting the model it runs on.