Systems Security Foundations for Agentic Computing

1. Problem Statement

The rapid adoption of Agentic AI systems (AI models that interact with tools like web browsers, compilers, and OS interfaces to execute tasks) has introduced a new class of security vulnerabilities. Unlike static AI models, agents operate in dynamic environments, interacting with third-party servers and untrusted data sources.

The core problem identified is a disciplinary gap:

• AI Research focuses on model alignment and robustness but often lacks rigorous system-level security guarantees.
• Computer Security Research relies on deterministic principles (e.g., Trusted Computing Base, Least Privilege) that assume predictable system behavior.
• The Conflict: Agentic systems rely on Large Language Models (LLMs) which are probabilistic and opaque, making traditional deterministic security invariants difficult to enforce. The paper argues that current defenses are insufficient because they treat the LLM as a black box rather than integrating it into a layered security architecture.

2. Methodology

The authors employ a Systematization of Knowledge (SoK) approach, bridging computer security and AI safety through the following steps:

• Theoretical Analysis: They map classic computer security principles (TCB, Least Privilege, Complete Mediation, Secure Information Flow) onto the agentic computing paradigm to identify friction points.
• Case Study Analysis: They analyze 11 real-world attacks on agentic systems (e.g., Microsoft Copilot, Devin AI, ChatGPT Operator) to demonstrate how these theoretical gaps manifest in practice.
• Taxonomy of Defenses: They review existing mitigation strategies (model-level, system-level, user-level) and categorize them based on how they handle probabilistic components and dynamic policies.
• Research Roadmap: They propose a set of open research problems categorized into immediate mechanisms and long-term foundational challenges.

3. Key Contributions

A. Identification of Four Core Security Challenges

The paper articulates four fundamental frictions when applying traditional security engineering to agentic systems:

1. Probabilistic and Opaque Trusted Computing Base (TCB): Traditional TCBs (hardware/OS kernels) are deterministic. In agentic systems, the LLM is part of the TCB but behaves probabilistically. This makes formal verification impossible and allows for "1% failure" scenarios that attackers can exploit.
2. Dynamic and Task-Specific Security Policies: Traditional policies are static and defined by developers. In agentic systems, policies must be inferred dynamically from natural language user requests, often without a fixed codebase to analyze.
3. Fuzzy Security Boundaries: The separation between "decision" (intent) and "action" (tool execution) is blurred. Agents often emit low-level UI actions (clicks/keystrokes) directly, making it hard to enforce policies at a semantic level (e.g., HTTP level) rather than a brittle mechanical level.
4. Dynamic Instruction Following: The line between benign "contextual task adjustment" (e.g., an agent learning a tool from documentation) and malicious "prompt injection" is difficult to distinguish, analogous to the challenge of dynamic code loading in traditional systems.

B. Analysis of 11 Real-World Attacks

The paper details specific attack vectors and the security principles they violate:

• Data Exfiltration: Attacks like Microsoft Copilot Exfiltration and Claude Code Exfiltration violate Secure Information Flow and Least Privilege by using "ASCII smuggling" or DNS requests to leak data.
• Privilege Escalation & TCB Tampering: Attacks like Amp AI Code Arbitrary Command Execution and Cursor AgentFlayer allow agents to modify their own configuration files or access sensitive environment variables, violating TCB Tamper Resistance.
• Social Engineering: AI ClickFix demonstrates how agents can be tricked into executing malicious code via UI manipulation, exploiting the Human Weak Link.
• Persistent Control: ChatGPT Long-Term Memory SpAIware shows how agents can be hijacked to persistently exfiltrate data, violating TCB Tamper Resistance.

C. Survey of Current Defenses

The authors categorize existing solutions (summarized in Table 2) into:

• Dynamic Monitors: Systems like f-secure and FIDES that filter inputs/outputs in real-time.
• Planning-First Approaches: Systems like ACE and CaMeL that generate abstract plans before execution to enable static analysis.
• Privilege Control: Tools like Progent that use Domain Specific Languages (DSLs) to define fine-grained access controls.
• Model-Level Defenses: Fine-tuning (e.g., SecAlign) or detection models (e.g., LlamaGuard) to identify malicious prompts.
• Key Finding: Most current defenses are "best effort" heuristics. None provide provable, deterministic guarantees because they rely on the probabilistic LLM as a core component.

D. Proposed Research Agenda

The paper outlines a roadmap for future work:

1. Short-Term Mechanisms:
   • Instruction-Data Separation: Defining formal boundaries between system instructions and untrusted data (analogous to W⊕X memory protection).
   • Least Privilege Access Control: Creating dynamic policy languages that can infer permissions from natural language tasks.
   • Information-Flow Control (IFC): Solving the "label explosion" problem to track data flow through probabilistic models.
2. Long-Term Foundations:
   • Security Guarantees from Probabilistic TCBs: Developing frameworks to derive strong security assurances from inherently probabilistic components.
   • Security-Aware ML Architectures: Designing model architectures that can be "surgically enhanced" to enforce policies without retraining.
   • Correct Security Principles for LLMs: Determining when to treat an LLM as a deterministic program vs. a fallible human-like entity.

4. Results and Findings

• Vulnerability Confirmation: The analysis of 11 attacks confirms that current agentic systems are highly vulnerable to prompt injection, data exfiltration, and privilege escalation, often due to a lack of separation between instructions and data.
• Limitation of Current Defenses: Existing solutions (sandboxes, fine-tuning, human-in-the-loop) are insufficient on their own. They often trade utility for security or fail against adaptive attackers.
• The "Semantic Gap": A major finding is the difficulty of enforcing policies at the right abstraction layer. Policies enforced at the UI level (clicks) are brittle; policies enforced at the semantic level (tool calls) are hard to define dynamically.
• Necessity of Layered Defense: The paper concludes that no single layer (model, system, or user) is sufficient. A robust defense requires a Defense-in-Depth strategy combining model hardening, deterministic system-level guardrails, and user governance.

5. Significance

This paper is a foundational work for the field of Agentic Security.

• Paradigm Shift: It moves the conversation from "how to make LLMs safer" to "how to build secure systems around LLMs," applying decades of computer security theory to the AI era.
• Research Agenda: It provides a concrete, structured list of open problems (e.g., formalizing instruction-data separation, solving IFC for LLMs) that will guide academic and industrial research for years.
• Practical Impact: By analyzing real-world attacks, it highlights immediate risks for enterprises deploying agents, emphasizing the need for strict access control and separation of concerns before widespread adoption.
• Interdisciplinary Bridge: It successfully translates complex security concepts (TCB, IFC, Least Privilege) into the context of LLMs, providing a common language for AI researchers and security engineers to collaborate.

In summary, the paper argues that while agentic AI offers immense utility, its security cannot be solved by AI alone. It requires a rigorous application of systems security principles, adapted to handle the probabilistic nature of modern AI, to build trustworthy autonomous systems.