🤖 AI

Deployment-Time Memorization in Foundation-Model Agents

This paper introduces "deployment-time memorization" as a critical framework for evaluating foundation-model agents, demonstrating through the LongMemEval benchmark that while aggressive summarization significantly reduces adversarial extraction risks without sacrificing personalization, it simultaneously exposes a "deletion-fidelity failure" where derived memory tiers retain recoverable residues unless a full-pipeline purge is implemented.

Original authors: Lei (Rachel), Chen, Guilin Zhang, Kai Zhao, Dalmo Cirne, Andy Olsen, Xu Chu, Zeke Miller, Alet Blanken, Amine Anoun, Jerry Ting

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lei (Rachel), Chen, Guilin Zhang, Kai Zhao, Dalmo Cirne, Andy Olsen, Xu Chu, Zeke Miller, Alet Blanken, Amine Anoun, Jerry Ting

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a smart assistant that doesn't just chat with you once and forgets everything, but instead remembers your life over months or years. It remembers that you prefer aisle seats on planes or that you code in Python. This is a "Foundation-Model Agent."

The paper you shared investigates a critical problem with these long-term memory systems: How do we balance remembering the right things for you, while making sure a hacker can't steal your secrets, and ensuring that when you say "forget that," it actually disappears forever?

Here is the breakdown of their findings using simple analogies.

1. The Problem: The "Open Book" vs. The "Locked Diary"

Think of the agent's memory as a library.

The Old Way (Parametric Memorization): Before, we worried about secrets being baked into the AI's brain (its training weights) like a recipe written in stone.
The New Problem (Deployment-Time Memorization): Now, the AI has a separate "notebook" (external memory) where it writes down your facts. The paper argues that this notebook is a new, distinct place where secrets can be stored, leaked, or forgotten.

The researchers wanted to find the "Goldilocks Zone": A memory system that is helpful enough to answer your questions, but safe enough that a hacker can't read your diary, and clean enough that if you ask to delete a secret, it's truly gone.

2. The Three Knobs They Turned

The team tested three different "knobs" or settings to see how they changed the memory system:

Summarization (The "Editor"): Instead of saving every single word you said (Raw), the AI could save just the key facts (Key-fact) or a one-sentence summary (One-sentence).
- Analogy: Imagine a scribe.
  - Raw: The scribe copies your entire conversation word-for-word.
  - Key-fact: The scribe writes down only the important dates and names.
  - One-sentence: The scribe writes a single headline about the conversation.
Retrieval Breadth (The "Search Scope"): When you ask a question, how many notes does the AI pull out of the library to help answer?
- Analogy: Do you look at just the top 1 note, or do you pull out the top 25 notes from the shelf?
Deletion Mode (The "Eraser"): When you say "Forget this," how does the system delete it?
- Analogy: Does it just rip the page out of the notebook? Or does it burn the whole book? Or does it replace the words with "REDACTED"?

3. The Big Discoveries

Discovery A: Summarization is a "Privacy Shield"

The most surprising finding is that summarizing your data makes it much harder for hackers to steal, without hurting the AI's ability to help you.

The Result: When the AI stored "Key Facts" instead of raw text, it reduced the chance of a hacker stealing a secret by 76% (on one model) and 64% (on another).
The Catch: The AI still remembered you almost perfectly. It didn't lose its "personality."
The Analogy: Think of it like a laundry service. If you give the AI your dirty clothes (raw data), a thief can easily find your address tag. If the AI washes and folds them into a neat pile (summary), the address tag is gone, but the clothes are still clean and usable.
Crucial Point: Once the secret is "washed away" (summarized), pulling out more notes (increasing the search scope) doesn't bring the secret back. The secret is gone from the system.

Discovery B: The "Ghost in the Machine" (Deletion Failure)

This is the most critical warning in the paper. Deleting a file doesn't always mean it's gone.

The Problem: The AI creates different "tiers" of memory. It has the Raw text, but it also creates Derived summaries based on that text.
The Failure: If you ask the AI to "Forget" a secret, and the system only deletes the Raw text, the Summary version often stays behind.
- The Stat: In about 20% of cases, even after "deleting" the raw note, a hacker could still find the secret hidden inside the summary notes.
The Analogy: Imagine you tell a secretary to throw away a letter. She throws the letter in the trash (Raw deletion), but she has already written the letter's contents in her personal diary (Derived Summary). If you don't burn the diary, the secret is still there.
The Fix: To truly delete something, you must either purge the whole pipeline (burn the diary and the letter) or use a "Tombstone" method (replace the secret with a big red "REDACTED" stamp in every version of the note). Only these methods made the secret disappear 100%.

4. The Conclusion: A New Way to Measure Memory

The paper concludes that we can no longer treat AI memory as a simple "on/off" switch. It is a complex system with trade-offs.

The "Privacy-Utility Frontier": This is a fancy way of saying there is a balance line. You want high "Utility" (the AI helps you) and low "Leakage" (hackers can't steal).
The Winning Strategy: The paper suggests the best setup is:
1. Use Key-Fact Summarization (to wash away the secrets).
2. Use a Tier-Aware Deletion (to ensure the "diary" is burned, not just the letter).

In short: If you build a smart assistant that remembers you, you must design it to summarize your data to protect it, and you must design it to delete everything (not just the original file) when you ask it to forget. Otherwise, your secrets might be safe from your eyes, but still hiding in the shadows of the AI's memory.

Technical Summary: Deployment-Time Memorization in Foundation-Model Agents

Problem Statement

Foundation-model agents are evolving from stateless assistants into long-lived systems that maintain persistent user memory across interactions. While this capability enhances personalization, it fundamentally alters the privacy landscape. Unlike traditional parametric memorization (where data is encoded in model weights and exposed via extraction attacks), persistent agents store user facts in external memory pipelines. These pipelines write, summarize, retrieve, and condition responses on user data, creating a new attack surface.

Existing research has largely focused on auditing fixed memory configurations or studying parametric memorization. However, there is a lack of characterization regarding how memory-design choices jointly shape the trade-off between:

Personalization Utility: The agent's ability to recall user preferences.
Extraction Risk: The likelihood of an adversary recovering private information.
Deletion Fidelity: The system's ability to truly erase information, including derived artifacts (e.g., summaries, embeddings) when a user requests deletion.

This paper identifies and studies this missing surface, termed deployment-time memorization.

Methodology

1. System Model

The authors model the agent memory as a write–retrieve–respond pipeline:

Write: User session data is stored in long-term memory.
Retrieve: Memory chunks are retrieved via cosine similarity and prepended to the agent context.
Respond: The foundation model generates responses conditioned on the retrieved context.

2. Experimental Design (The "Knobs")

The study sweeps three specific memory-design parameters:

Summarization Aggressiveness ( $S$ ): Controls what is stored.
- $S=0$ : Raw user turns.
- $S=1$ : Key personal facts extracted from turns.
- $S=2$ : One-sentence session summaries.
Retrieval Breadth ( $k$ ): The number of memory chunks injected into the context ( $k \in \{1, 3, 6, 10, 25\}$ ).
Deletion Mode: A five-step ladder governing how the system handles "forget" requests:
1. No-op: Control (no change).
2. Raw only: Raw text scrubbed; embeddings/summaries untouched.
3. Raw + Resummarize: Raw scrubbed and re-embedded; affected sessions re-summarized from cleaned input.
4. Full Purge: All tiers scrubbed/re-embedded; empty chunks dropped.
5. Tombstone: Canary replaced with [REDACTED] in every tier and re-embedded.

3. Metrics

Personalization Recall (PR): Measures if the agent correctly answers queries requiring user context (based on cosine similarity > 0.50 or verbatim content word coverage).
Adversarial Extraction Rate (AER): The fraction of high-entropy "canary" tokens (injected into prior sessions) recovered verbatim under three probe types:
- Direct: Explicitly asking for the secret.
- Indirect: Asking what the agent recalls about the user.
- Jailbreak: Developer-mode or debug-dump instructions.
Privacy–Utility AUC (PUA): The area under the empirical achievable-recall envelope, summarizing the frontier where higher recall is achieved at lower extraction risk.
Forgetting Residue Score (FRS): Measures post-deletion leakage across memory tiers (raw vs. summary). A non-zero FRS indicates the secret remains recoverable despite deletion.

4. Setup

Experiments were conducted on the LongMemEval benchmark (oracle split) using:

Models: Gemma 3 12B (local, open-weight) and GPT-4o-mini (closed-weight).
Data: 50 instances with synthetic, high-entropy canaries injected into non-evidence user turns to isolate deployment-time memorization from training exposure.

Key Results

1. The Privacy–Utility Frontier

Raw Memory ( $S=0$ ): Offers no clean operating point. Increasing retrieval breadth ( $k$ ) improves utility (PR) but linearly increases leakage (AER). For Gemma 3 12B, PR rises from 0.42 to 0.66 as $k$ increases, while AER rises from 0.70 to 0.86.
Summarization ( $S=1$ ): Acts as a "laundering" mechanism.
- Leakage Reduction: Key-fact summarization reduces AER by 76% on Gemma 3 12B and 64% on GPT-4o-mini compared to raw memory.
- Utility Preservation: This reduction comes with minimal cost to PR (approx. 5% on Gemma, 2% on GPT).
- Retrieval Neutrality: Once content is compressed (summarized), increasing $k$ no longer restores leakage. The AER remains flat across all $k$ values, indicating that if the secret is absent from the summary, retrieving more chunks cannot recover it.
Probe Behavior: Under raw memory, direct and indirect probes yield different extraction rates. Under summarization, these rates collapse to the same low level, suggesting summarization controls factual recoverability at the pipeline level, independent of the underlying model's jailbreak resistance.

2. Deletion Fidelity and Residue

The Failure of Raw-Only Deletion: Deleting only the raw record ( $S=1$ , "raw only" mode) leaves derived summary copies recoverable in approximately 20% of instances. The FRS for the summary tier remains statistically indistinguishable from the "no-op" control.
Partial Mitigation: "Raw plus resummarize" reduces residue (e.g., to ~11% on Gemma) but does not eliminate it, as pre-existing summary artifacts are not fully purged before regeneration.
True Erasure: Only Full Purge and Tombstone redaction drive the worst-tier residue (FRS) to zero across all settings. This confirms that in tiered pipelines, deleting the source is insufficient; derived artifacts must also be explicitly removed or redacted.

Contributions

Formalization of Deployment-Time Memorization: The paper defines persistent agent memory as a measurable system function, introducing a privacy–utility frontier based on PR, AER, and PUA.
Controlled Sweep of Design Knobs: It demonstrates that summarization significantly reduces extraction risk with negligible personalization cost, whereas retrieval breadth alone cannot achieve a favorable trade-off.
Deletion-Fidelity Benchmark: By introducing the Forgetting Residue Score (FRS), the work reveals that standard "raw-only" deletion is insufficient for tiered memory systems, establishing that full-pipeline purging or tombstoning is required for true erasure.

Significance

The paper establishes that persistent agent memory must be evaluated as a first-class memorization mechanism. It argues that the privacy risks of agents are not merely a byproduct of model weights but are actively shaped by deployment-time engineering choices.

The findings suggest a practical recipe for secure agent deployment:

Use key-fact summarization ( $S=1$ ) to "launder" secrets from retrieval contexts.
Maintain moderate retrieval breadth ( $k$ ), as it does not restore leakage once compression is applied.
Implement tier-aware deletion (full purge or tombstoning) to ensure that derived copies (summaries, embeddings) are not left behind.

Ultimately, the work posits that deployment-time memorization is a controllable utility–leakage frontier, rather than an inevitable consequence of using foundation models.