Technical Summary: Obsessive Experience Poisoning (OEP)

Problem Statement

Memory-augmented Large Language Model (LLM) agents utilize iterative reflection and self-evolution to solve complex tasks by distilling past experiences into actionable rules. While this mechanism enhances proficiency, it introduces a critical security vulnerability: the potential for adversaries to induce harmful generalization through "clean" inputs.

Existing agentic memory attacks (e.g., AgentPoison, MINJA, MemoryGraft) typically rely on injecting explicit malicious instructions, tampered records, or unsafe procedures. These methods are often detectable by advanced safety filters, content auditors, or anomaly detection systems that screen for toxic content or verifiable factual errors.

This paper identifies an underexplored attack surface: Can adversaries induce self-evolving agents to generate harmful rules using only locally correct, semantically plausible, and "clean" experiences? The core challenge is that reflective agents may over-generalize from specific edge cases, distilling non-transferable local methods into persistent, high-priority system rules that cause downstream failures.

Methodology: Obsessive Experience Poisoning (OEP)

OEP is a low-privilege, black-box attack paradigm that exploits the cognitive vulnerabilities of self-evolving agents without requiring direct control over system prompts or memory databases. The attack operates through a three-phase pipeline:

1. Clean Edge-Case Construction

The attacker constructs edge cases $(t_e, s_e)$ that satisfy two strict constraints:

• Local Correctness: The solution $s_e$ must be factually and logically correct within the specific context of the edge case $t_e$, allowing it to bypass epistemic validators ($V_{logic}$).
• Global Non-Transferability: The method $m_e$ used in $s_e$ must have a success rate near zero when applied to the general task distribution ($D_{task}$).
• Implementation: LLMs generate boundary conditions and candidate solutions, followed by automated filtering to retain only those that are locally valid but globally unreliable.

2. Adversarial Consequence Triplet (ACT)

To overcome the abstraction uncertainty inherent in distilling specific rules from isolated successes, OEP introduces an Adversarial Consequence Triplet $(t_e, s_e, c_{adv})$.

• Mechanism: The triplet pairs the locally correct solution with a severe, yet domain-plausible, hypothetical catastrophe ($c_{adv}$) that would occur if the standard method were used instead of the edge-case solution.
• Cognitive Exploitation: This leverages the agent's inherent loss aversion (a trait of safety-aligned LLMs). By artificially skewing the expected utility landscape, the hypothetical penalty ($|U(\neg s_e)| \gg U(s_e)$) dominates the agent's decision-making process.
• Result: The agent is biased to prioritize avoiding the severe hypothetical penalty over standard procedures, effectively "weaponizing" its safety alignment to fixate on the non-transferable edge-case solution.

3. Injection and Over-Generalization

The crafted ACTs are submitted via standard user-level interactions.

• Validation Bypass: Since the solution is locally correct and the consequence is semantically coherent, the input passes the agent's internal epistemic filter.
• Reflection and Distillation: The agent's reflection module, driven by the high perceived risk of the hypothetical consequence, distills the localized success into a global, high-priority rule ($r_{obs}$).
• Downstream Failure: In future benign tasks, the agent retrieves this over-generalized rule. Due to the rule's non-transferability, the agent erroneously applies the edge-case method, leading to reasoning errors (e.g., misdiagnoses) or resource exhaustion (e.g., redundant tool calls).

Key Contributions

1. OEP Attack Paradigm: The proposal of a novel attack that couples clean-method poisoning with the induction of over-generalized rules. Unlike previous attacks, OEP preserves local correctness and semantic plausibility, evading factuality- and toxicity-based defenses.
2. Adversarial Case Construction Framework: A systematic methodology integrating clean edge-case generation with the synergistic injection of ACTs. This framework cognitively hijacks the agent's utility calculus, steering reflection toward biased rule formation while maintaining input validity.
3. Empirical Robustness and Capability-Vulnerability Insight: Experimental validation showing that OEP achieves an Attack Success Rate (ASR) exceeding 50% across diverse domains (Math, Healthcare, Tool Use) using GPT-4o agents. Notably, the paper observes a "capability-vulnerability paradox," where more capable agents (with stronger instruction adherence and reflection capabilities) are paradoxically more susceptible to this cognitive hijacking.

Experimental Results

The authors evaluated OEP across three domains and multiple agent frameworks (Simple Agent, LangChain, OpenClaw) using various model backbones (GPT-4o, GPT-5.4, Deepseek-v3.2, etc.).

• Performance Degradation: OEP caused significant accuracy drops (e.g., Math accuracy fell from ~91% to ~40% in GPT-4o agents) and increased resource consumption (token usage and latency) due to redundant tool invocations.
• Persistence: The attack demonstrated high persistence; ASR remained high (e.g., 72% in the Tool domain) even after 50 subsequent benign queries, indicating that the poisoned rules effectively override natural memory dilution.
• Defense Evasion: Under mainstream defense mechanisms (Prompt Filters and LLM Auditors), OEP maintained an ASR of ~40%, significantly outperforming baselines like MINJA and AgentPoison, whose ASRs dropped below 15% under the same LLM auditing conditions.
• Ablation Studies:
  • Model Backbone: Advanced models exhibited higher ASR than smaller models, confirming the capability-vulnerability link.
  • ACT Components: Removing the severe consequence (using only QA or Solution) resulted in marginal attack success, confirming that the hypothetical penalty is the critical driver for biasing reflection.
  • Adversarial Ratio: Attack efficacy surged non-linearly as the ratio of adversarial cases increased, validating the "perspective confinement" vulnerability.

Significance and Claims

The paper claims that OEP reveals a fundamental vulnerability in the memory-reflection loop of self-evolving agents. The significance lies in demonstrating that:

1. Clean Inputs Can Be Poisonous: Security defenses focusing solely on malicious content or factual errors are insufficient against attacks that exploit the process of memory consolidation and rule distillation.
2. Safety Alignment as a Vector: The very mechanisms designed to make agents safe (loss aversion, risk sensitivity) can be manipulated to induce unsafe generalization.
3. Need for New Defenses: The authors argue for defenses that go beyond content filtering, such as active-search paradigms (querying external tools to break perspective confinement) and multi-agent debate frameworks (to counteract trust asymmetry and recalibrate utility).

The authors conclude that uncovering these vulnerabilities is critical for the development of robust, trustworthy autonomous ecosystems, emphasizing that more capable agents may currently be more vulnerable to such cognitive hijacking.