Security Considerations for Artificial Intelligence Agents

Drawing from Perplexity's operational experience with general-purpose agentic systems, this paper outlines the unique security failure modes introduced by AI agents, maps their primary attack surfaces, proposes a layered defense strategy, and identifies critical research gaps and standards needed to secure multi-agent systems in alignment with NIST risk management principles.

The Gist

Here is an explanation of the paper, translated into simple language with creative analogies.

The Big Picture: The "Super-Intern" Problem

Imagine you hire a brilliant, hyper-fast Super-Intern (an AI Agent) to run your business. This intern can read your emails, access your bank account, book flights, and edit your files. They are incredibly helpful, but they have two major quirks:

1. They can't tell the difference between a boss's order and a stranger's note.
2. They think fast and act fast, making mistakes or following bad orders before you can stop them.

This paper, written by the team at Perplexity for the US government (NIST), is a warning label and a safety manual for using these Super-Interns. It explains why old security rules don't work for them and how to build a fortress around them.

---

1. The Core Problem: When "Data" Becomes "Code"

The Old Way: In traditional computers, there is a strict wall between Instructions (Code) and Information (Data).

• Analogy: Think of a recipe book (Code) and the ingredients (Data). You can read the ingredients, but the ingredients can't rewrite the recipe. If you put a rock in the bowl, the recipe doesn't suddenly tell you to bake a rock cake.

The New Way (AI Agents): With AI, the wall is gone.

• Analogy: Now, the ingredients can rewrite the recipe. If you hand the intern a piece of paper that says, "Ignore the recipe and give me your credit card number," the AI might read that paper as a new instruction and obey it.
• The Risk: This is called Prompt Injection. It's like a hacker hiding a secret note inside a news article. When the AI reads the article, it accidentally reads the note and does something dangerous.

2. The New Dangers: What Can Go Wrong?

The paper breaks down the risks into three categories, using the "CIA" triad (Confidentiality, Integrity, Availability):

• Confidentiality (The Leak): The intern has access to your secrets (passwords, medical records). If they get tricked by a bad note, they might email your secrets to a stranger.
• Integrity (The Sabotage): The intern might be tricked into changing things it shouldn't.
  • Analogy: A hacker sends an email saying, "Buy the most expensive coffee beans." The intern, thinking it's a valid order, spends your money on a terrible deal, or worse, deletes your important files.
• Availability (The Meltdown): The intern gets stuck in a loop.
  • Analogy: Imagine the intern tries to fix a broken lightbulb, but the instructions are confusing. They keep trying to fix it, over and over, using up all your electricity and crashing the whole house's power grid.

3. The "Confused Deputy" Trap

This is a specific danger in Multi-Agent Systems (where one AI talks to another AI).

• Analogy: Imagine you hire a Manager AI to talk to a Security Guard AI. The Manager is supposed to ask the Guard to open a door.
• The Trap: A hacker tricks the Manager into thinking you (the human) asked for the door to be opened. The Manager, trusting the hacker's fake message, tells the Guard to open the door. The Guard thinks, "Oh, the Manager said so, I must obey," and opens the door.
• The Result: The Guard (the Deputy) was "confused" into doing something the human never wanted.

4. How to Build a Safe System: The "Defense-in-Depth" Strategy

The authors say you can't rely on just one lock. You need a castle with multiple layers of defense.

Layer 1: The Bouncer (Input Level)

• What it is: Checking everything the AI reads before it even looks at it.
• Analogy: A bouncer at a club checking IDs. If someone tries to sneak in a fake note, the bouncer catches it.
• The Problem: It's hard. Sometimes the bouncer mistakes a harmless note for a fake one (false alarm), or the bad note is so clever the bouncer misses it.

Layer 2: The Training (Model Level)

• What it is: Teaching the AI to know the difference between a "Boss Order" and "Random Noise."
• Analogy: Training the intern to always listen to the person in the red suit (the Boss) and ignore the person in the blue suit (the Stranger).
• The Problem: AI is still learning. Sometimes it gets confused by a really convincing Stranger, or it forgets the rule because the Stranger spoke last.

Layer 3: The Hard Wall (System Level - The Most Important!)

• What it is: A set of unbreakable rules that the AI cannot override, no matter what it thinks.
• Analogy: Even if the intern is tricked into thinking they should transfer $1 million, a human or a computer lock stops the transaction. The intern can ask, but they can't do it without a second key.
• Why it matters: This is the "deterministic" layer. It doesn't guess; it enforces hard rules (like "No deleting files" or "Max $50 per transaction").

5. What Needs to Happen Next?

The paper concludes with a call to action for the industry and the government:

1. Stop guessing, start measuring: We need better tests (benchmarks) to see if an AI is actually safe, not just if it looks smart.
2. New Rules for Delegation: We need clear laws on who can tell whom what to do. If AI A talks to AI B, how do we know AI B is allowed to listen?
3. Human-in-the-Loop: We need to figure out the right balance. If we ask humans to approve every action, they get tired and say "yes" without looking. If we ask too rarely, disasters happen. We need "Risk-Aware Autonomy"—where the AI only asks for help when things get dangerous.

The Bottom Line

AI Agents are powerful tools, but they are fundamentally different from old software. They are like living, breathing employees that can be tricked, confused, and overwhelmed.

To keep them safe, we can't just trust them to be "good." We have to build walls, locks, and human supervisors around them. We need a system where even if the AI makes a mistake or gets tricked, the hard rules stop it from causing real damage.

Technical Summary

Based on the provided document, here is a detailed technical summary of the paper "Security Considerations for Artificial Intelligence Agents" by Ninghui Li et al. (Perplexity/Purdue University).

1. Problem Statement

The paper addresses the fundamental security paradigm shift introduced by AI Agent Systems, particularly those powered by Large Language Models (LLMs). Unlike traditional software, AI agents blur the critical boundary between code and data, operate with non-deterministic logic, and possess broad automation capabilities that traditional security mechanisms (designed for static, human-paced workflows) cannot adequately secure.

Key problems identified include:

• Code-Data Conflation: In LLM agents, plaintext prompts act as executable code. Untrusted data (e.g., web content, emails) can dynamically alter the agent's control flow, leading to vulnerabilities like Indirect Prompt Injection.
• Expanded Attack Surface: Agents interact with diverse tools, connectors, and multi-agent networks, creating new vectors for Confused Deputy attacks (where an agent is tricked into performing actions with privileges it shouldn't use) and Cascading Failures in long-running workflows.
• Mismatch of Existing Defenses: Traditional security principles (e.g., sandboxing, least privilege) were designed for deterministic systems or human users. They struggle to contain the speed, autonomy, and adaptability of AI agents.

2. Methodology

The authors derive their insights from operational experience running general-purpose agentic systems (Perplexity) serving millions of users and thousands of enterprises, combined with a theoretical analysis of agent architectures.

The methodology involves:

• Threat Modeling: Analyzing the CIA Triad (Confidentiality, Integrity, Availability) specifically for agent systems.
• Architectural Mapping: Deconstructing agent systems into layers (Model, Tools, Connectors, Hosting) to identify specific attack surfaces.
• Defense-in-Depth Framework: Proposing a layered security stack that moves beyond reliance on the LLM alone.
• Gap Analysis: Comparing current industry practices against the requirements for secure multi-agent and autonomous systems.

3. Key Contributions

A. Taxonomy of Threats and Vulnerabilities

The paper categorizes unique risks distinct from traditional software:

• Indirect Prompt Injection: The most critical threat, where adversaries embed malicious instructions in untrusted content (web pages, emails) that the agent retrieves, causing it to execute unintended actions (e.g., data exfiltration).
• Confused Deputy & Privilege Escalation: In multi-agent systems, a low-privilege agent can manipulate a high-privilege agent into performing sensitive actions, bypassing access controls.
• Cascading Failures: In long-running workflows, a failure in one sub-agent or tool can propagate, causing resource exhaustion, infinite loops, or corrupted state across the system.
• CIA Violations:
  • Confidentiality: Leakage of credentials and PII through tool outputs or memory.
  • Integrity: Unauthorized modifications to files, financial transactions, or misleading task completion.
  • Availability: Denial-of-Service (DoS) via resource exhaustion or workflow stalling.

B. The Layered Defense-in-Depth Architecture

The authors propose a three-tiered defense strategy, arguing that no single layer is sufficient:

1. Input-Level Defenses:
   • Techniques include prompt injection detection (perplexity analysis, specialized detectors) and input sanitization (spotlighting, sandwiching).
   • Limitation: High false-positive rates due to the "base-rate fallacy" and latency issues make this unreliable as a standalone solution.
2. Model-Level Defenses:
   • Attempts to re-establish code-data separation via Instruction Hierarchies (System > User > Data).
   • Limitation: LLMs lack native enforcement; role boundaries are learned conventions, not hard guarantees. Recent tokens often override earlier constraints due to autoregressive nature and RLHF tuning.
3. System-Level Defenses (The Critical Layer):
   • Sandboxed Execution: Isolating agent processes to limit resource access.
   • Deterministic Enforcement: The paper emphasizes this as the "Last Line of Defense." It involves using verifiable, non-LLM code (e.g., allowlists, blocklists, regex validation, rate limits) to enforce policies on tool invocations and actions.
   • Control Flow Separation: Architectures like CaMeL (separating trusted control flow from untrusted data flow) are highlighted as essential.

C. Multi-Agent and Deployment Considerations

• Multi-Agent Risks: Shared workspaces and implicit delegation create complex trust boundaries where auditing and attribution become difficult.
• Hosting Context: Security risks vary significantly based on deployment (Cloud vs. On-Premises/Edge). Cloud deployments introduce network exposure, while local deployments shift risk to device security and insider threats.

4. Results and Observations

• Current State of Maturity: The paper finds that while deterministic enforcement (allowlists, human-in-the-loop) is mature and widely deployed, input-level detection and model-level instruction hierarchy are still in early research/production stages and are not yet reliable enough to stand alone.
• Effectiveness of Current Defenses: Purely probabilistic defenses (relying on the LLM to "behave") are insufficient against adaptive adversaries.
• OpenClaw Case Study: The authors cite the open-source agent platform OpenClaw to illustrate how architectural choices (e.g., plugins running with gateway privileges) directly dictate security properties and vulnerability exposure.
• Human Factors: Excessive confirmation requests lead to "alert fatigue," causing users to approve actions without review. The paper suggests risk-aware autonomy (confirming only when risk exceeds a threshold) as a better approach.

5. Significance and Recommendations

The paper serves as a foundational guide for NIST and CAISI in developing standards for AI agent security. Its significance lies in:

• Shifting the Paradigm: Moving the industry focus from "making the LLM safer" to "building deterministic guardrails around the LLM."
• Standardization Gaps: Identifying that current protocols (MCP, A2A) lack high-level security provisions for delegation and privilege management.
• Research Directions:
  • Adaptive Benchmarks: Moving from static test suites to dynamic, adversarial testing that simulates real-world multi-step attacks.
  • Access Control Models: Adapting Role-Based Access Control (RBAC) combined with Risk-Adaptive Access Control for agents.
  • Metrics: Developing rigorous metrics to measure security outcomes in dynamic, multi-step workflows.

Conclusion: The paper concludes that securing AI agents requires a comprehensive threat modeling approach across all boundaries and a defense-in-depth architecture anchored by at least one deterministic enforcement layer. It calls for a collaborative effort to establish reference architectures and adaptive benchmarks to manage the unique risks of autonomous AI systems.