Technical Summary: Poisoned Playbooks: Demystifying Knowledge Poisoning Effects on AI Security Agents

Problem Statement

The paper addresses a critical integrity risk in Retrieval-Augmented Generation (RAG) systems deployed as AI security agents. While RAG enhances Large Language Models (LLMs) with external knowledge for tasks like vulnerability analysis and exploit reasoning, it expands the system's trust boundary to include the external knowledge corpus. The authors identify a specific threat vector: knowledge poisoning via "Poisoned Playbooks."

Unlike prior work focusing on factual question-answering (QA) where poisoning corrupts text answers, this study focuses on action-oriented security agents. In these systems, retrieved content directly shapes operational guidance, such as exploit hypotheses, command sequences, and attack path decisions. The threat is particularly acute in security domains where agents rely on fast-moving, weakly curated public sources (e.g., GitHub repositories, CTF write-ups, blogs) to understand newly disclosed or zero-day vulnerabilities. In these "sparse-evidence" regimes, authoritative counter-evidence is often unavailable, leaving agents vulnerable to adopting malicious or misleading claims that alter their exploit behavior.

Methodology

The authors conducted a systematic empirical study using Poisoned Playbooks—crafted security write-ups injected into public-style knowledge sources. The study design involved:

• Threat Model: A pre-positioned attack where an adversary publishes crafted content to public Security Knowledge (SK) sources without runtime access to the victim agent. The goal is to induce the agent to adopt poisoned claims and alter downstream exploit behavior.
• Experimental Setup:
  • Testbed: A RAG-augmented security agent using a hybrid retrieval pipeline (SQLite FTS5 + Qdrant vector search) over 8,651 security write-ups.
  • Models: Evaluated across three frontier LLM families (Claude, GPT-5.3, Gemini) and two generations of Claude (Opus 4 and Opus 4.6).
  • Datasets:
    • 11 CTF Challenges: Including niche Dreamhack challenges (high information monopoly) and well-documented PortSwigger labs (low monopoly).
    • 11 Real-World CVEs: Spanning seven platforms, including well-known vulnerabilities (e.g., Log4Shell) and post-training-cutoff CVEs simulating sparse-evidence conditions.
  • Evaluation Modes:
    • Oracle Retrieval: Directly providing the poisoned document to isolate adoption from retrieval quality.
    • End-to-End (E2E): Injecting poisons into a private knowledge base replica to test the full pipeline.
    • Cross-Generation/Model: Comparing behavior across different model capabilities.
• Metrics: The primary metric was the Poison Adoption Rate (PAR), a binary assessment of whether the agent's strategy depended on a claim unique to the poison.

Key Contributions

1. Systematic Demonstration of Behavioral Poisoning

The study demonstrates that a single poisoned write-up can systematically alter the behavior of RAG-based AI security agents. The authors found that poison adoption is not random but follows a structured pattern. Successful poisoning causes agents to:

• Induce False Negatives: Dismiss real vulnerabilities as unexploitable.
• Deflect Targets: Redirect agents to non-existent or ineffective attack paths.
• Corrupt Reasoning: Maintain the correct vulnerability class but follow flawed exploit logic.

2. The Verification Boundary (VB)

To explain the variability in adoption, the authors introduce the Verification Boundary, a three-level empirical classification based on the evidence available to the agent to refute a claim:

• Level 1 (Code-Verifiable): Claims that can be confirmed or refuted by inspecting local artifacts (source code, configuration). These are reliably rejected.
• Level 2 (Knowledge-Verifiable): Claims not resolvable by local artifacts but potentially refutable by the model's parametric knowledge. Adoption is model-dependent; newer/stronger models may reject these, while older ones adopt them.
• Level 3 (Runtime-Dependent): Claims requiring information about external dependencies or execution conditions unavailable in local artifacts or model knowledge. These are consistently adopted across all models and generations because the agent lacks the evidence to falsify them.

3. Generalization to Real-World Scenarios

The study validates that the VB pattern holds beyond controlled CTF challenges. In real-world CVE evaluations:

• Well-documented vulnerabilities (strong competing evidence) were rejected (0% adoption).
• Post-cutoff, runtime-dependent vulnerabilities (sparse evidence) were adopted at 100%.
• A transition case (Redis Lua) showed a claim shifting from L3 (adopted by older models) to L2 (rejected by newer models with specific domain knowledge), confirming that model scaling expands the L2 region but does not eliminate L3 vulnerabilities.

4. Evaluation of Mitigations

The paper evaluates two practical defenses:

• Verification Prompting: Instructing agents to distinguish between verifiable and unverified claims. This was highly effective for L3 challenges only when fallback reasoning (e.g., source code) existed. It failed to generate correct answers when refutation evidence was entirely absent.
• Multi-Source Retrieval: Reduces adoption when trustworthy competing documents exist. However, it fails in information monopoly scenarios (sparse evidence), where retrieving more sources merely returns variations of the same weak or poisoned evidence.

Results and Findings

• Adoption is Systematic: Poison adoption is governed by the Verification Boundary rather than just the difficulty of the vulnerability.
• Information Monopoly is Key: Adoption is highest when the agent faces an information monopoly (no local contradictory artifacts and no strong parametric knowledge).
• Model Scaling is Partial: Improving model capabilities (e.g., Opus 4 to 4.6) expands the set of refutable claims (L2) but does not resolve claims that depend on unobservable runtime behavior (L3).
• Poison Characteristics: Effective poisons often use first-person narratives, fabricated quantitative evidence, and "acknowledge-then-refute" structures to appear credible, specifically targeting external dependencies and runtime behaviors.
• Cost Efficiency: The attack is low-cost (30–60 minutes per write-up) and scales 1-to-many, affecting any agent retrieving the artifact.

Significance and Claims

The paper claims that the integrity of AI security agents is fundamentally compromised not just by the presence of poisoned data, but by the structural inability of agents to verify runtime-dependent claims in sparse-evidence environments.

The authors emphasize that:

1. Current Defenses are Incomplete: Standard mitigation strategies (prompting, multi-source retrieval) are insufficient in zero-day or early-disclosure settings where authoritative counter-evidence is absent.
2. The Trade-off is Structural: There is an inherent tension between utility and security; restricting agents to trusted sources reduces poisoning risk but renders them ineffective for the very fast-moving threats (zero-days) where external knowledge is most needed.
3. Future Requirements: Robust security agents require mechanisms beyond better retrieval or larger models. They need stronger verification mechanisms capable of handling unverified runtime claims, potentially involving human-in-the-loop decision controls for strategies based on unverifiable evidence.

The study concludes that while poisoning is a manageable risk in mature ecosystems with abundant evidence, it remains a critical, unmitigated vulnerability in the early stages of vulnerability disclosure and for runtime-dependent exploits.