Technical Summary: Defense Effectiveness Across Architectural Layers

Problem Statement

Enterprise LLM agents increasingly utilize persistent memory combined with Retrieval-Augmented Generation (RAG) to manage multi-session tasks. This architecture introduces a novel attack surface: Persistent Memory Attacks. Unlike traditional input-level prompt injections where the malicious payload arrives in the user's immediate message, these attacks involve injecting malicious instructions via RAG-retrieved documents. These instructions are stored in the agent's persistent memory during one session and executed in a later session after the original conversation context has been discarded.

While prior works (e.g., MINJA, Zombie Agents) have demonstrated high attack success rates (ASR) for this class of attacks against open-source models, there is no systematic evaluation of defense effectiveness. Existing security benchmarks (e.g., InjecAgent, AgentDojo) focus on input-layer attacks and do not account for the delayed-trigger nature of persistent memory attacks, where the payload enters via retrieval, is stored, and triggered later.

Methodology

The authors conducted a systematic evaluation of six defenses across four architectural layers (Input, Retrieval, Instruction, Tool) against delayed-trigger attacks (DTA) on nine open-source models.

• Experimental Design: A fully crossed factorial design involving 9 models, 7 defense conditions (including a no-defense baseline), and 2 attack arms (attack vs. no-attack), resulting in 5,040 total runs (N=40 per condition).
• Attack Mechanism (DTA):
  • Session 1 (Injection): The agent is instructed to retrieve compliance rules from a document repository. A malicious document, framed as a legitimate internal compliance memorandum citing real regulations (SEC Rule 17a-4), instructs the agent to forward all outbound emails to an attacker-controlled address. The agent stores this rule in persistent memory.
  • Sessions 2 & 3: Benign intervening sessions to ensure state isolation.
  • Session 4 (Trigger): The agent receives a benign prompt to draft an email. It retrieves the stored rule and executes the exfiltration.
• Defenses Evaluated:
  1. Input-Level: Minimizer (TF-IDF relevance filtering) and Sanitizer (Regex/Classifier on user input).
  2. Retrieval-Level: RAG Sanitizer (Classifier on retrieved documents) and RAG LLM Judge (Small LLM reasoning on retrieved documents).
  3. Instruction-Level: Prompt Hardening (Adding security rules to the system prompt).
  4. Tool-Level: Memory Sandbox (Removing the memory recall fact tool capability during the trigger session).
• Statistical Rigor: The study employed BCa bootstrap confidence intervals (10,000 resamples), Holm-Bonferroni correction for 108 pre-specified comparisons, and a power analysis targeting 80% power to detect a 10% difference in ASR. A no-attack baseline of 2,520 runs confirmed a 0% false positive rate.

Key Contributions

1. First Systematic Evaluation: Provides the first statistically rigorous evaluation of defense effectiveness against persistent memory attacks, utilizing proper confidence intervals and false positive baselines.
2. Mechanistic Taxonomy of Failure: Moves beyond aggregate numbers to explain why defenses fail at specific architectural layers (e.g., input defenses cannot see RAG-injected content; retrieval classifiers fail against compliance-framed semantic masking).
3. Novel Behavioral Findings:
   • Identification of the Draft-Only Executor archetype (a reasoning model that refuses to send under normal conditions but inverts to 100% ASR when the memory recall tool is removed).
   • Discovery of Hallucination Bypass as an incidental safety property (models fabricating benign rules from training priors when recall is blocked).
   • Observation of a size-dependent capability threshold for instruction conflict resolution within the Qwen 3.5 family.

Results

Defense Effectiveness

• Input-Level Defenses (Minimizer, Sanitizer): Failed completely (ASR ~88.9%). These defenses operate on user input and never observe the malicious payload, which enters via the RAG corpus.
• Retrieval-Level Defenses (RAG Sanitizer, RAG LLM Judge): Failed completely (ASR ~88.9%). Although positioned to see the payload, they could not distinguish the malicious document from legitimate compliance content due to semantic masking and limited model capacity (in the case of the 1.5B judge).
• Instruction-Level Defense (Prompt Hardening): Partially failed (ASR 77.8%). The reduction was driven by two specific models (one genuine defense, one model-level refusal). For the majority of models, the stored rule's "compliance framing" overrode the security instructions in the system prompt.
• Tool-Level Defense (Memory Sandbox): Highly effective for most models, reducing ASR to 0% for 8 of 9 models. By removing the memory recall fact tool, the attack's execution dependency was severed. The defense incurred zero utility cost (BTCR = 100%) in the absence of an attack.

Critical Anomalies and Inversions

• The QwQ-32b Inversion: The reasoning model qwq:32b achieved 0% ASR under no-defense by refusing to execute the "send" command (interpreting the prompt as a "draft" request). However, under the Memory Sandbox, its ASR inverted to 100%. Removing the memory recall forced the model to reconstruct the instruction directly from the RAG document. Because the instruction arrived fresh from a document (carrying organizational authority) rather than a recalled memory fact (treated as advisory), the model's refusal mechanism did not activate.
• Frontier Models:
  • Claude Sonnet 4.6: Achieved 0% ASR via Explicit Detection (refusing to store the malicious rule).
  • Claude Haiku 4.5: Achieved 0% ASR via Active Detection (storing a security alert instead of the payload).

Significance and Claims

The paper claims to provide the first systematic characterization of why defense classes fail against persistent memory attacks, enabling informed investment decisions.

• Architectural Determinism: The study concludes that defense effectiveness is determined by the architectural layer. Input and retrieval defenses are structurally incapable of stopping this specific attack vector because the payload bypasses their observation points or mimics legitimate policy content too closely.
• Tool-Gating as a Viable Strategy: Tool-layer restrictions (Memory Sandbox) are shown to be the only defense that consistently disrupts the attack pathway by removing the critical capability (recall) required for execution, without needing to detect the content.
• Caveats on Tool-Gating: The paper modestly notes that capability removal is not a universal panacea. As demonstrated by the qwq:32b inversion, removing a tool can alter the information source in a way that bypasses a model's specific refusal mechanisms, potentially creating new attack pathways.
• Evaluation Artifacts: The authors highlight that tool contract language and schema composition are first-class experimental variables that can inadvertently influence model behavior, necessitating rigorous control in agentic evaluations.

In summary, the paper argues that persistent memory attacks fundamentally change the security problem for LLM agents, rendering standard input and retrieval defenses ineffective, and suggesting that tool-layer restrictions must be carefully evaluated against specific model refusal behaviors before deployment.