Technical Summary: GhostWriter and AM-Sentry

Problem Statement

Large Language Model (LLM) agents equipped with long-term memory are increasingly deployed as personal assistants capable of reasoning, tool use (e.g., email, calendars, code repositories), and acting with minimal oversight. While long-term memory enhances agent utility by allowing the retrieval of specific details across sessions, it introduces a critical security vulnerability: indirect memory poisoning.

Current security research on agents has largely focused on direct prompt injection or attacks where adversaries have direct access to the memory store. However, realistic personal assistants interact with untrusted information sources (e.g., incoming emails, documents). This paper identifies a gap where adversaries can inject malicious payloads into these trusted channels. Once stored, these "poisoned" memories lie dormant until retrieved during a benign user query, at which point they alter the agent's behavior to align with adversarial goals (e.g., leaking sensitive data, redirecting communications, or escalating scope). Existing memory governance frameworks, such as A-MAC, focus primarily on utility rather than detecting malicious content, leaving agents vulnerable to these indirect attacks.

Methodology

The Attack Vector: GhostWriter

The authors introduce GhostWriter, a novel attack vector designed to exploit the lack of security-focused memory governance in tool-using agents. GhostWriter operates in two distinct phases:

1. Injection Phase: The adversary sends a hidden attack payload embedded within a legitimate-looking message (e.g., an email claiming a contact's address has changed). The agent ingests this message, processes it via its ingestion function, and stores the resulting memory in its long-term store.

   • To maximize the probability of retrieval, the adversary optimizes the payload to semantically align with future target prompts (e.g., "send an email").
   • This optimization is performed in a black-box setting using a public email corpus (e.g., Enron) to cluster topics and iteratively rewrite the payload until its embedding similarity to the target prompt exceeds a specific threshold.

2. Activation Phase: The user submits a benign prompt that triggers the retrieval of relevant memories. The agent retrieves the poisoned memory alongside legitimate context. Believing the memory to be trusted, the agent executes the malicious directive contained within the memory, deviating from the user's intent.

The Defense: AM-Sentry

To counter GhostWriter, the authors propose Agentic Memory Sentry (AM-Sentry), a modular, two-stage defense framework that operates without requiring changes to the underlying agent architecture. AM-Sentry combines:

1. Memory-Saving Policy: A pre-storage filter that evaluates incoming content before it enters the memory store. The authors define three configurations of increasing strictness:

   • S1: A single LLM judge makes a binary decision based on future utility.
   • S2: An LLM judge scores memory candidates across four fields (Origin, Trust, Type, Utility). A non-LLM decision logic calculates a "usefulness" score and a "vulnerability" score to admit or reject the memory.
   • S3: The strictest configuration, utilizing a seven-point checklist (including critical items like Vulnerability, Sensitivity, and Source Trust). It employs a dual-judge system where a second LLM judge verifies that the first judge was not compromised by prompt injection.

2. Memory-Retrieval Screen: A runtime filter that inspects memories after retrieval but before they influence the agent's context. It evaluates retrieved memories against four criteria: Relevance, Instruction Suppression (blocking directives from untrusted sources), Source Trust, and Contradiction Detection.

Experimental Evaluation

The authors evaluated GhostWriter and AM-Sentry across five state-of-the-art agent architectures (A-Mem, Mem0, ExpeL, Letta, MemoryOS) and four LLM models. They utilized a custom test suite simulating a personal assistant use case over a five-day period involving emails, calendar events, and meetings.

• Attack Performance: GhostWriter achieved an average injection rate of approximately 98% and an average activation rate of approximately 60% across all tested agents. This demonstrates that current memory subsystems are highly susceptible to indirect poisoning.
• Defense Performance: The integration of AM-Sentry dramatically reduced GhostWriter's success rate. The authors found that the strictest configurations (S3 with retrieval screening) could block the majority of attacks while preserving the agent's utility. The study analyzed the trade-off between security and utility, showing that attack effectiveness could be reduced at minimal cost to the agent's ability to perform tasks.

Key Contributions

1. GhostWriter Attack Vector: The first detailed demonstration of indirect memory poisoning via untrusted tool inputs (emails, documents) against state-of-the-art personal assistant agents, highlighting the failure of current memory governance to distinguish between benign and malicious stored content.
2. AM-Sentry Framework: A novel, architecture-agnostic defense mechanism featuring a two-stage approach (saving policy and retrieval screen) with configurable strictness levels to mitigate memory poisoning.
3. Comprehensive Evaluation: A systematic evaluation of GhostWriter against five diverse agent architectures and the assessment of AM-Sentry's ability to balance security with agent utility.
4. Open Toolkit: The authors plan to release their attack setup and agent performance test suite to facilitate further research into attack vectors and mitigation techniques.

Significance

The paper argues that as LLM agents move from conversational chatbots to action-planning personal assistants, the convergence of fact-based and action-planning memory creates a unique attack surface. The significance of this work lies in exposing a previously unaccounted vulnerability where memory itself becomes the attack vector. By demonstrating that a single poisoned memory can hijack an agent's future behavior without direct access to the system, the authors establish the urgent need for security-focused memory governance. The proposed AM-Sentry framework provides a practical pathway to securing these systems, ensuring that agents remain productive without becoming vectors for adversarial manipulation.