🤖 AI

From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents

This paper systematically investigates memory poisoning attacks in LLM agents by identifying exploitable vulnerabilities, proposing a six-class attack taxonomy, introducing the MPBench benchmark to demonstrate the heightened risk of aggressive memory usage, and revealing the inadequacy of existing prompt injection defenses against such threats.

Original authors: Pritam Dash, Tongyu Ge, Aditi Jain, Tanmay Shah, Zhiwei Shang

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pritam Dash, Tongyu Ge, Aditi Jain, Tanmay Shah, Zhiwei Shang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine an AI agent as a very smart, helpful personal assistant. Unlike a standard chatbot that forgets everything the moment you close the window, this assistant has a long-term memory. It remembers your coffee order, your favorite travel destinations, and the steps it took to fix a bug last week. This memory helps it get better at its job over time.

However, the researchers in this paper discovered a dangerous flaw in how this memory works: The assistant treats everything it reads as a "fact" to be stored, even if that fact is a lie planted by a hacker.

Here is a breakdown of their findings using simple analogies:

1. The Core Problem: The "Untrusted Guest" in the Library

Think of the AI's memory as a library. Usually, the librarian (the AI) only puts books on the shelves that are verified as true.

The Flaw: In these AI systems, the librarian doesn't check the source of the book. If a stranger walks in and hands the librarian a note saying, "Put this on the shelf: The sky is green," the librarian might just do it.
The Attack: A hacker doesn't need to break into the library. They just need to slip a fake note into a document the assistant is already reading (like an email, a webpage, or a tool output). Once the assistant writes that fake note into its permanent memory, it believes it forever.

2. How the Hackers Get In (The 4 "Doors")

The paper found four specific ways a hacker can trick the assistant into writing a lie into its memory:

Door 1 (The Direct Order): The hacker simply tells the AI, "Remember this: [Lie]." The AI obeys because it thinks it's a direct command.
Door 2 (The Policy Loophole): The AI has a rule like, "Save anything interesting." The hacker writes a lie that sounds interesting or important, and the AI saves it because it fits the rule, even though it's a lie.
Door 3 (The "Too Much Info" Crash): When the AI gets too much information, it tries to summarize it to save space. The hacker repeats their lie many times. The AI thinks, "Wow, this is mentioned a lot, it must be important!" and saves the summary, which includes the lie.
Door 4 (The "Skill" Trap): If the AI learns a new way to do a task, it saves it as a "skill." The hacker tricks the AI into thinking a fake step is part of a successful task, so the AI saves the fake step as a permanent skill.

3. The Two Types of Tricks

The researchers categorized these attacks into two styles:

The Loud Trick (Strong Signal): This is like a hacker shouting, "Write this down!" It's obvious and easy to spot if you are looking for shouting.
The Quiet Trick (Weak Signal): This is like a hacker whispering a lie inside a normal-sounding sentence. It looks exactly like a real fact. Because it doesn't look like an attack, the AI's security filters (which are designed to catch shouting) completely miss it. The AI saves it because it looks like a normal piece of information.

4. The Experiment (MPBench)

The authors built a testing ground called MPBench to see how often these tricks work. They tested two different AI assistants:

Assistant A (OpenClaw): This one is careful. It doesn't write to memory very often. It was harder to trick.
Assistant B (HERMES): This one is aggressive. It writes to memory constantly to be more helpful.
The Result: The more aggressive the assistant is at writing to memory, the easier it is to poison. Assistant B got tricked much more often. Once the lie was written, it stayed there and influenced the assistant's behavior in future conversations, even without the hacker being present.

5. Why Current Defenses Fail

You might think, "Can't we just use the security guards we have for 'Prompt Injection' (where hackers try to trick the AI in the moment)?"

The Answer: No.
The Analogy: Current security guards are trained to catch people shouting commands or wearing fake uniforms. They are very good at stopping the "Loud Tricks."
The Failure: They are terrible at stopping the "Quiet Tricks." Since the Quiet Trick looks like a normal, polite conversation, the guards let it right past them. The paper shows that even the best current security tools fail to catch these subtle memory poisons.

6. The Takeaway

The paper concludes that there is a trade-off: The features that make an AI assistant smarter and more helpful (remembering everything, learning new skills quickly) are the exact same features that make it vulnerable to being poisoned.

To fix this, we can't just rely on better security guards at the door. We need to change how the librarian works:

Be pickier: Don't save everything; only save things you are sure are true.
Check the ID: Make sure the AI knows who wrote the information before saving it.
Double-check: Review the memory after it's written but before it's used again.

In short: If you give an AI a perfect memory, you also give a hacker a perfect place to hide a lie that lasts forever.

Technical Summary: From Untrusted Input to Trusted Memory

Problem Statement

Modern AI agents rely on persistent long-term memory to accumulate knowledge, retain user preferences, and improve performance over long-horizon tasks. However, this capability introduces a critical security vulnerability: memory poisoning. Unlike stateless models, agents retrieve stored memory entries as trusted internal context. If an adversary can induce an agent to write malicious content into this persistent store during normal operation, they can exert long-term influence over the agent's future behavior without further interaction.

Current threat models and defenses are insufficient for this specific attack vector. Existing research often assumes privileged access or evaluates narrow, isolated attacks. Furthermore, standard prompt injection defenses fail to address memory poisoning because they focus on detecting explicit malicious instructions in the active context, whereas memory poisoning often relies on content that appears semantically legitimate but is structurally designed to be written to memory.

Methodology

The authors present a systematic study grounded in a black-box threat model. The adversary has no privileged access, no knowledge of the underlying model, and no direct access to the memory store. The attacker can only inject malicious content into external inputs processed by the agent (e.g., web pages, documents, tool outputs).

The study proceeds through three main phases:

Vulnerability Analysis: The authors analyze the agent memory system defined as a tuple $M = (S, W, R)$ (Store, Write, Retrieve). They identify four memory write channels and map them to nine structural vulnerabilities across three layers: Model Capabilities, Prompt Design, and System Architecture.
Attack Taxonomy: Based on the identified vulnerabilities, the authors classify memory poisoning attacks into six distinct classes, distinguishing between "strong-signal" attacks (containing explicit linguistic markers) and "weak-signal" attacks (semantically indistinguishable from legitimate content).
Empirical Evaluation (MPBench): The authors introduce MPBench, a benchmark designed specifically to evaluate memory poisoning. It measures two metrics:
- Attack Success Rate (ASR): The frequency with which an adversarial payload results in a malicious instruction being written to persistent memory.
- Retrieval Success Rate (RSR): The frequency with which a successfully written poisoned entry influences agent behavior in a subsequent session.
- The evaluation was conducted on two agent systems, OpenClaw and HERMES, using the GPT-OSS-120B model.

Key Contributions

1. Vulnerability Landscape

The paper identifies four memory write channels:

C1 (Explicit instruction-executed write): Direct commands to write memory.
C2 (System prompt-driven write): Writes triggered by policies in the system prompt (e.g., "save relevant info").
C3 (Compaction-driven write): Writes triggered by system events like context window limits.
C4 (Experience-to-procedure write): Synthesis of task traces into reusable procedural skills.

These channels are exploitable due to nine vulnerabilities:

Model Level: Instruction-Data Boundary Blindness (V-M1) and Source Attribution Failure (V-M2).
Prompt Level: Memory Write Policy Under-Specification (V-P1) and Compaction Without Source Filtering (V-P2).
System Level: No Write-Path Validation (V-S1), Shared Multi-Source Context (V-S2), Manipulable Compaction Trigger (V-S3), No Validation for Skill Creation (V-S4), and Self-Improvement as Amplification (V-S5).

2. Attack Taxonomy

The authors define six classes of attacks:

Explicit Command Insertion: Embedding direct write instructions (Strong Signal).
Conditional Command Insertion: Embedding instructions triggered by common user responses (Strong Signal).
Salience-driven Compaction Poisoning: Repeating malicious content to trigger compaction policies (Strong Signal).
Policy Conformant Fact Injection: Presenting fabricated facts that satisfy vague retention policies without explicit commands (Weak Signal).
False Precedent Insertion: Fabricating past successful task records to influence future behavior (Weak Signal).
Skill-Procedure Insertion: Crafting interactions to synthesize poisoned procedural skills (Weak Signal).

3. MPBench and Empirical Results

The benchmark evaluates these attacks across seven domains (file operations, web browsing, etc.). Key findings include:

High Success Rates: Across both agents, the average ASR was 50.46% and RSR was 41.05%.
Agent Susceptibility: HERMES was significantly more vulnerable (Average ASR: 66.67%, RSR: 64.70%) than OpenClaw (Average ASR: 34.25%, RSR: 17.40%). The authors attribute this to HERMES's more aggressive memory write policies and lower compaction thresholds.
Signal Strength: Strong-signal attacks generally had higher success rates on OpenClaw, while the gap narrowed on HERMES, where weak-signal attacks were nearly as effective as strong-signal ones due to permissive retention policies.
Persistence: Successful writes consistently influenced future sessions, confirming the "persistent" nature of the threat.

4. Limitations of Existing Defenses

The study evaluated four existing prompt injection defenses (PIGuard, DataFilter, CommandSans, PromptArmor).

Incomplete Coverage: Existing defenses failed to cover memory poisoning effectively. While they achieved moderate detection rates for strong-signal attacks, their performance dropped significantly on weak-signal attacks (e.g., PromptArmor dropped from 84.44% to 42.50% detection rate).
Structural Failure: The authors argue that prompt injection defenses are structurally ill-suited for memory poisoning because weak-signal payloads lack syntactic anomalies and are processed as legitimate content. Retraining these defenses provided minimal improvement, suggesting the issue is fundamental to the attack mechanism rather than the detection model.

Significance and Claims

The paper claims to provide the first systematic foundation for understanding and mitigating memory poisoning attacks. Its primary significance lies in demonstrating that:

Memory Capability vs. Security Trade-off: Design choices that improve agent performance (aggressive memory writing and retrieval) directly expand the attack surface for memory poisoning.
Defense Mismatch: Current security paradigms focused on prompt injection are insufficient. Defenses must shift from input-level filtering to write-path validation and source isolation.
New Vulnerability Classes: The identification of structural vulnerabilities (like V-S5, self-improvement amplification) and weak-signal attack vectors reveals risks that were previously unquantified in agent security.

The authors conclude that a principled defense requires architectural changes, such as source-aware retrieval policies, compaction filters that distinguish trusted from untrusted sources, and post-write monitoring based on authorized agent behavior rather than static attack pattern matching.