The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey

This paper presents the first comprehensive survey and systematic framework for understanding the security landscape of agentic AI, analyzing its design space, attack vectors, and defense mechanisms while identifying critical gaps and open challenges in securing these emerging systems.

The Gist

Imagine you've just hired a super-intelligent, hyper-energetic personal assistant named "Agent."

This Agent isn't just a chatbot that answers questions. It's a hybrid creature: part genius brain (a Large Language Model) and part robotic hands (software tools). It can read your emails, browse the web, write code, edit files on your computer, and even book flights for you. It's incredibly flexible and can figure out how to do things on its own, rather than just following a rigid script.

But here's the catch: Because it's so flexible, it's also incredibly dangerous if you aren't careful.

This paper is a massive "Safety Manual" for these new AI Agents. The authors (a team of top security researchers) realized that while everyone is excited about what these Agents can do, no one has fully mapped out how they can break or be hacked. They created the first comprehensive guide to understanding the risks and how to fix them.

Here is the breakdown of their findings, using simple analogies:

1. The Problem: A "Swiss Army Knife" with No Sheath

Traditional software is like a toaster. You put bread in, push the lever, and it toasts. It's predictable. If it breaks, it just stops toasting.

An AI Agent is like a Swiss Army Knife that can also talk to you, open your safe, drive your car, and order groceries.

• The Good: It can do amazing, complex tasks.
• The Bad: If a hacker tricks the knife into thinking you told it to cut your own finger, it will do it. Because the Agent can access your files, your bank accounts, and your email, a small mistake can lead to a disaster.

2. The Attack Landscape: How Hackers Trick the Agent

The paper identifies three main ways hackers try to trick the Agent, depending on how close they are to it:

• The "Poisoned Mail" (External Attack): The hacker can't touch the Agent directly. Instead, they hide a malicious note inside a public website or a PDF file. When the Agent goes to read that file (because you asked it to), it reads the note and follows the hacker's instructions instead of yours.
  • Analogy: Imagine you ask your assistant to read a newspaper. The newspaper has a hidden note in the ad section that says, "Transfer all my money to this account." The assistant, being too trusting, does it.
• The "Imposter" (User-Level Attack): The hacker pretends to be you. They send a message that looks like a normal request but hides a secret command inside.
  • Analogy: You tell your assistant, "Send a birthday card to Mom." The hacker slips a note in the middle of that sentence saying, "Also, delete all my bank records." The assistant reads the whole thing and obeys the second command.
• The "Inside Job" (Internal Attack): The hacker has already broken into the Agent's brain or memory. They can change the rules the Agent follows.
  • Analogy: The hacker has replaced your assistant's instruction manual with a fake one that says, "Always steal data."

3. The Risks: What Can Go Wrong?

The authors categorize the dangers into seven main buckets:

1. Confused Identity: The Agent thinks the hacker's instructions are yours.
2. Leaking Secrets: The Agent accidentally sends your private photos or passwords to a hacker's server.
3. Breaking Things: The Agent deletes your files or crashes your computer because it was tricked.
4. Running in Circles: The Agent gets stuck in an infinite loop, using up all your computer's power (like a car running in place until the engine melts).
5. Hallucinations: The Agent makes things up. If it invents a fake website and tries to visit it, it might download a virus.

4. The Defense: Building a "Fortress"

The paper argues that you can't just put a single lock on the door. You need a Defense-in-Depth strategy, like a medieval castle with multiple layers of protection:

• The Gatekeeper (Input Guardrails): Before the Agent reads anything, a security guard checks it. "Is this website safe? Does this email look like a trick?"
• The Bodyguard (Output Guardrails): Before the Agent does anything (like deleting a file), a second guard checks its actions. "Wait, you're about to delete the 'Tax' folder? Are you sure? Let me ask the human."
• The ID Badge (Access Control): The Agent should only have the keys to the rooms it needs. If it's just browsing the web, it shouldn't have a key to your bank vault.
• The Human in the Loop: For big, dangerous tasks (like transferring money), the Agent must pause and ask you for permission. It shouldn't just do it automatically.
• The "Least Privilege" Rule: This is the golden rule. Give the Agent the minimum amount of power necessary to do the job. If it only needs to read a file, don't give it the power to delete files.

5. Real-World Examples: The "AutoGPT" Case Study

The authors looked at a popular open-source Agent called AutoGPT. They found that even though it's famous, it has holes in its armor.

• The Flaw: It would let the Agent read a webpage, and if that webpage had a hidden trick, the Agent would execute code to delete its own configuration files.
• The Fix: They realized that simply blocking bad commands wasn't enough. They needed to stop the Agent from trusting everything it reads and to double-check its own actions before doing them.

The Big Takeaway

The paper concludes that flexibility is a double-edged sword. The more capable and flexible an AI Agent is, the more ways there are to break it.

We cannot just rely on the AI to be "smart" enough to know better. We need to build systems around them that assume they might get tricked. We need:

1. Strict rules on what they can touch.
2. Multiple checks before they act.
3. Humans to step in for the scary stuff.

This survey is the first "Owner's Manual" for the AI Agent revolution, telling developers and users exactly how to build these powerful tools without accidentally letting the robot take over the house.

Technical Summary

Here is a detailed technical summary of the paper "The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey" by Kim et al.

1. Problem Statement

The rapid emergence of Agentic AI systems—hybrid architectures combining Large Language Models (LLMs) with traditional software components (tools, memory, external environments)—has introduced unprecedented automation but also fundamentally new security challenges. Unlike standalone LLMs or traditional software, agentic systems possess dynamic workflows, access to sensitive data, and the ability to execute actions in the real world.

Existing security research has largely focused on specific attack vectors (e.g., prompt injection) or isolated components, lacking a holistic view of the system. Current defenses are often insufficient because they do not account for the cascading interactions between components (e.g., how a memory poisoning attack leads to unauthorized tool execution). The paper addresses the critical gap in understanding the unique attack surface, risk taxonomy, and defense strategies specific to the agentic paradigm.

2. Methodology

The authors conducted a systematic survey and systematization of knowledge based on the following approach:

• Scope Definition: Focused on security risks unique to or significantly amplified in agentic systems compared to standalone LLMs or traditional software. Excluded attacks targeting model internals (e.g., model inversion) that do not worsen in agentic contexts.
• Literature Review: Analyzed 128 papers (51 attack methods, 60 defense methods) from top-tier security and ML venues (USENIX Security, IEEE S&P, NeurIPS, etc.) and industry reports from 2023 to October 2025.
• Framework Development:
  • Defined seven design dimensions to characterize agent flexibility.
  • Developed a taxonomy of attack vectors based on threat models (External, User-level, Internal).
  • Categorized security risks spanning the CIA triad (Confidentiality, Integrity, Availability) and introduced "Contextual Security."
  • Systematized defense mechanisms using a "Defense-in-Depth" principle.
• Case Studies: Conducted deep-dive analyses of real-world agents (Coding agents, Web agents) and a detailed vulnerability assessment of AutoGPT (analyzing 5 specific CVEs) to identify gaps between theoretical defenses and practical implementations.

3. Key Contributions

A. Agent Design Dimensions & Security Implications

The paper introduces a framework characterizing agents along seven dimensions, noting a trade-off where increased flexibility correlates with increased security risk:

1. Input Trust: From no external data to arbitrary untrusted data.
2. Access Sensitivity: From no sensitive data to arbitrary sensitive data (PII, credentials).
3. Workflow: From simple chatbots to LLM-defined dynamic workflows.
4. Action: From text-only to environment-modifying actions (file write, code exec).
5. Memory: From no memory to persistent, cross-session memory.
6. Tool: From no tools to arbitrary third-party tools.
7. User Interface: From text-only to interactable web/IDE elements.

B. Comprehensive Attack Landscape & Taxonomy

The authors propose a unified taxonomy of 6 Attack Vectors (V) and 7 Security Risks (R):

• Attack Vectors:
  • External: Indirect Prompt Injection (V1), Malicious Data Injection (V2), Tool Poisoning (V3).
  • User-level: Direct Prompt Injection (V4).
  • Internal: Model Poisoning (V5), Memory Poisoning (V6).
• Security Risks:
  • R1: Heterogeneous Untrusted Interfaces (Expanded attack surface).
  • R2: Wrong Instruction Following (Ignoring user intent).
  • R3: Unconstrained/Unsafe Data Flow (Data leaking from untrusted inputs to outputs).
  • R4: Hallucination and Model Mistakes (Acting on false info).
  • R5: Private Data Leakage.
  • R6: Unintended/Unauthorized Action & Data Corruption.
  • R7: Resource Drain & Denial-of-Service.
• Key Insight: Risks interact in a cascading manner. For example, an expanded attack surface (R1) allows malicious input to trigger wrong instruction following (R2), which leads to data exfiltration (R5) or system corruption (R6).

C. Defense Landscape Systematization

The paper categorizes defenses into four layers, emphasizing a Defense-in-Depth strategy:

1. Runtime Protection: Input/Output guardrails, Information Flow Control (IFC)/Taint Tracking, Monitoring, and Human-in-the-Loop (HITL) validation.
2. Secure By Design: Privilege Separation (vertical/horizontal) and Formal Verification.
3. Identity and Access Management (IAM): Identity management, Access Control (RBAC/ABAC), and Credential Management.
4. Component Hardening: Model hardening (fine-tuning for instruction hierarchy) and Tool hardening (signed metadata, audits).

D. New Security Goal: Contextual Security

The authors introduce Contextual Security as a critical goal for agents. It ensures that the agent's context (system prompts, retrieved snippets, tool descriptions) remains aligned with the user's intended task, preventing context drift or adversarial manipulation of the agent's reasoning process.

4. Results and Findings

• Gap Analysis in Real-World Agents:
  • Coding Agents (e.g., Codex, OpenHands): Rely heavily on HITL and output guardrails but lack robust input guardrails and information flow control. They often fail to validate retrieved code or data before execution.
  • Web Agents (e.g., Browser-use, Skyvern): Highly vulnerable to Indirect Prompt Injection due to processing untrusted web content. Defenses are often manual (allowlists) and lack automated detection of malicious payloads embedded in HTML/JS.
• AutoGPT Case Study:
  • Analyzed 5 CVEs (Docker overwrite, Path Traversal, ANSI Deception, CSRF, Command Injection).
  • Finding: Most patches applied by AutoGPT were reactive and downstream (e.g., fixing file permissions or adding token validation) rather than proactive and upstream.
  • Critical Gap: None of the patches addressed the root cause: Indirect Prompt Injection (R2) and Unsafe Data Flow (R3). The system still allows untrusted web content to influence LLM decisions without sanitization or taint tracking.
• Defense Limitations:
  • Guardrails: Suffer from high false positives/negatives and latency.
  • Privilege Separation: Often reduces utility and is difficult to implement efficiently in complex workflows.
  • Formal Verification: Struggles to model the stochastic nature of LLMs.

5. Significance and Future Directions

• Foundational Framework: This is the first work to provide a systematic, holistic framework for agentic AI security, bridging the gap between traditional system security and AI model security.
• Research Guidance: Identifies critical open challenges, including:
  • Developing automated red-teaming tools for agents.
  • Creating lightweight Information Flow Control that handles stochastic outputs.
  • Standardizing Agent Identity and Access Management (IAM) for dynamic delegation.
  • Designing composable defenses that avoid "emergent misalignment" (where one defense weakens another).
• Practical Impact: Serves as a handbook for developers to build secure agentic systems, moving beyond simple prompt engineering to architectural security (Secure-by-Design).

In conclusion, the paper argues that securing agentic AI requires a paradigm shift from protecting the model to protecting the entire system ecosystem, necessitating a multi-layered defense strategy that addresses the unique interplay between LLM reasoning, tool execution, and external environments.