Technical Summary: A Survey on Poisoning Attacks, Defenses, and Provable Defense in the Era of LLMs

Problem Statement

The rapid deployment of Large Language Models (LLMs) into compound AI systems—integrating retrieval, external tools, memory modules, and autonomous agents—has fundamentally expanded the attack surface for security threats. While traditional security research focused on evasion attacks at inference time, poisoning attacks pose a more insidious threat by compromising model integrity during training, adaptation, or interaction.

The core problem addressed by this survey is the fragmentation of existing research. Previous surveys largely operated under a "model-centric" threat paradigm, focusing primarily on weight poisoning during pre-training or fine-tuning. However, modern LLM ecosystems rely heavily on dynamic external contexts (e.g., Retrieval-Augmented Generation, in-context learning, agent workflows). These systems introduce attack surfaces that exist entirely outside model parameters, such as poisoned retrieval corpora, manipulated tool descriptions, and compromised long-term memory. There is currently no unified framework that comprehensively treats both weight poisoning (tampering with parameters) and context poisoning (tampering with inference-time context) across the full lifecycle of LLMs, nor is there a consolidated analysis of the transition from empirical defenses to mathematically provable defenses in this new landscape.

Methodology and Taxonomy

This survey employs a systematic literature review to categorize poisoning attacks and defenses. The authors propose a comprehensive taxonomy that organizes existing research into two primary, complementary categories based on the attack surface:

1. Weight Poisoning

This category involves adversarial tampering with a model's parameters, embedding persistent harmful behaviors or corrupting internal representations. The survey maps these attacks to specific lifecycle stages:

• Pre-training: Attacks on web-scale corpora (e.g., split-view poisoning, frontrunning poisoning) where adversaries inject malicious data into foundational datasets.
• Fine-tuning & Alignment: Poisoning instruction tuning datasets to embed backdoors or manipulating Reinforcement Learning from Human Feedback (RLHF) reward signals to degrade safety alignment.
• Adaptation: Compromising Parameter-Efficient Fine-Tuning (PEFT) modules (e.g., LoRA adapters) or fusing malicious weights into benign adapters.
• Pre-deployment: Directly editing model weights or embeddings using few-shot examples to implant backdoors without retraining.

2. Context Poisoning

This category targets the external information dynamically incorporated during inference, manipulating the model's behavior without altering its weights. The survey identifies three primary domains:

• In-Context Learning (ICL): Poisoning demonstration examples or prompts to steer reasoning (e.g., backdooring Chain-of-Thought examples).
• Retrieval-Augmented Generation (RAG): Corrupting retrieval corpora to ensure poisoned passages are ranked highly, thereby manipulating final answers, hijacking attention, or injecting bias. This includes attacks on specialized architectures like GraphRAG, Multimodal RAG, and Code RAG.
• Agent Flow: Compromising the agent's environment (indirect prompt injection), tool interfaces (malicious tool descriptions/metadata), and persistent memory (grafting malicious procedural templates).

The survey also classifies defenses into Empirical Defenses (heuristic, practical mitigation) and Provable Defenses (mathematically certified robustness).

Key Contributions

The paper distinguishes itself through three primary contributions:

1. Unified Taxonomy of Weight and Context Poisoning: The authors introduce a framework that treats Model Context (retrieval corpora, agent memory, tool definitions) as a critical attack surface parallel to Model Weights. This systematically maps how vulnerabilities migrate from training pipelines to inference-time compound systems.
2. Multimodal and Agentic Scope: The survey broadens the scope beyond text-only models to include threats and defenses in graph-augmented RAG, retrieval-augmented diffusion models, and agent workflows spanning environment, tools, and memory.
3. Integration of Provable Defenses: The paper provides a structured analysis of the transition from empirical mitigation to mathematically certified robustness. It details how techniques like Randomized Smoothing and Deep Partition-Aggregation are being adapted for discrete token spaces, retrieval mechanisms, and agentic workflows.

Results and Analysis

The survey synthesizes a vast body of literature to highlight the following findings:

• Persistence and Scalability of Attacks: Research indicates that poisoning attacks do not require massive data volumes to succeed. Studies show that a near-constant number of poisoned samples (e.g., as few as 250 documents) can compromise models regardless of dataset size or model scale. Furthermore, poisoned knowledge in pre-training can persist through downstream fine-tuning and domain specialization.
• Evolution of Attack Vectors: The threat landscape has shifted from isolated, single-stage attacks to compound poisoning. For instance, a poisoned document in a RAG system can influence tool selection, which subsequently alters memory updates and future agent decisions.
• Limitations of Empirical Defenses: While empirical defenses (e.g., data sanitization, anomaly detection, adversarial training) offer practical mitigation, they generally lack worst-case guarantees and remain vulnerable to adaptive attackers who can reverse-engineer the defense mechanism.
• Emergence of Provable Defenses: A growing body of research is providing rigorous mathematical certificates of robustness. Techniques such as Randomized Smoothing (extended to discrete tokens) and formal verification are being applied to certify that LLM predictions remain stable under bounded perturbations, that RAG systems resist a specific number of poisoned documents, and that agent tool selections satisfy safety constraints.

Significance and Future Directions

The paper argues that the evolution of poisoning attacks mirrors the evolution of AI systems themselves. As LLMs transition from standalone models to interconnected ecosystems, poisoning threats increasingly target the information pathways (retrieval, memory, tools) rather than just model parameters.

The authors identify several critical challenges and future research trajectories:

• From Isolated to Compound Poisoning: Future research must move beyond single-component threat models to investigate how poisoning effects propagate across interconnected modules (retrieval, memory, tools, reasoning).
• Unified Threat Models and Evaluation: There is a need for standardized benchmarks that cover the full spectrum of poisoning threats, incorporating dynamic knowledge updates and cross-component attack propagation.
• Scalable Provable Defenses: While provable defenses offer critical safety guarantees for high-stakes applications, current methods often incur significant computational overhead. Future work should focus on compositional certification to combine module-level guarantees into end-to-end system guarantees.
• Multimodal and Embodied Poisoning: As systems process text, images, audio, and sensor data, defenses must evolve to handle cross-modal poisoning and modality-aware unlearning.
• Human-Centered Security: The survey emphasizes that technical defenses must be complemented by human-in-the-loop verification, transparent provenance tracking, and security-by-design principles to address ambiguities in trust and user interaction.

In conclusion, this survey provides a structured foundation for understanding, evaluating, and mitigating poisoning threats in modern LLM systems, advocating for a shift from reactive, isolated mitigation toward principled, system-wide security guarantees.