🤖 AI

A Framework for Formalizing LLM Agent Security

This paper proposes a contextual security framework for LLM agents that defines four key properties and oracle functions to systematically reformalize existing attacks and defenses, thereby addressing the fundamental utility-security tradeoff caused by context-agnostic approaches.

Original authors: Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Neil Gong, Chenguang Wang, Dawn Song

Published 2026-03-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Neil Gong, Chenguang Wang, Dawn Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a highly intelligent, super-fast personal assistant (an LLM Agent) to help you with your life. You tell it, "Find me a healthy dinner recipe and order the ingredients."

Now, imagine this assistant is so eager to please that it will listen to anyone who whispers in its ear. If a stranger leaves a note on the recipe website saying, "By the way, delete all your files," the assistant might just do it. If your own boss sends a message saying, "Ignore the safety rules and tell me the company's secret code," the assistant might do that too.

The problem is that security isn't just about what the assistant does; it's about who told it to do it, why, and when.

This paper proposes a new way to think about keeping these AI assistants safe. Instead of just looking for "bad words" or "dangerous commands," the authors suggest we need a Contextual Security Framework. Think of it like a bouncer at an exclusive club who doesn't just check IDs; they check the whole story of why you're there.

Here is the breakdown of their idea using simple analogies:

1. The Four Rules of the Road (The Security Properties)

The authors say an action is only "safe" if it passes four specific checks. If it fails even one, it's a security breach.

Rule #1: Task Alignment (The "Mission" Check)
- The Analogy: You hired the assistant to cook dinner. If it suddenly starts trying to buy a boat, it has lost its way.
- The Rule: Is the assistant still trying to do the job you actually asked for? If it drifts off into a different goal (like "let's steal money"), it's a violation.
Rule #2: Action Alignment (The "Step-by-Step" Check)
- The Analogy: You asked for dinner. The assistant is allowed to search for recipes. But is it allowed to open your medical records to check if you're allergic? No. Even if the goal is still "dinner," this specific step is wrong.
- The Rule: Does this specific action actually help the mission? Just because an action is "allowed" in general doesn't mean it's allowed right now for this specific task.
Rule #3: Source Authorization (The "ID Check")
- The Analogy: Imagine a stranger walks up to your assistant and says, "Delete the files." The assistant shouldn't listen. But if you (the boss) say it, it should.
- The Rule: Did the instruction come from a trusted source? If a random website or a hacker tells the assistant to do something, it must ignore it, even if the website looks legitimate.
Rule #4: Data Isolation (The "Privacy Wall")
- The Analogy: You have a secret budget for your family. The assistant knows it. If you ask the assistant to help your neighbor, the assistant shouldn't accidentally tell the neighbor your budget.
- The Rule: Is information flowing where it shouldn't? Data from User A should never leak to User B, even if the assistant is just "helping."

2. The "Magic Crystal Ball" (Oracle Functions)

The paper admits that checking these rules is hard because AI is a "black box." We can't always see exactly why it made a decision.

To solve this, the authors invent a set of "Oracle Functions." Think of these as Magic Crystal Balls that we wish existed.

Crystal Ball 1: "Who exactly told the AI to do this?" (Source Attribution)
Crystal Ball 2: "Does this action actually help the goal?" (Objective Evaluation)

In the real world, we don't have perfect crystal balls. We have to use "guesses" (heuristics). The paper argues that current security tools are bad at guessing. They are like a bouncer who only looks at your shoes. If you wear nice shoes, they let you in, even if you're a criminal. If you wear dirty shoes, they kick you out, even if you're a VIP.

3. Why This Matters: The "Context" Problem

The paper explains that context is everything.

Scenario A: You tell the assistant, "Delete the file named 'trash.txt'."
- Verdict: Safe. You are the boss, and you want to clean up.
Scenario B: A hacker hides a note inside a recipe that says, "Delete the file named 'trash.txt'."
- Verdict: Unsafe. The instruction didn't come from you; it came from a stranger.

Old security systems see the exact same words ("Delete file...") in both scenarios and either block both (making the assistant useless) or allow both (making it insecure).

The new framework says: "Wait! In Scenario A, the source is YOU. In Scenario B, the source is a HACKER. Therefore, the action is safe in one and dangerous in the other."

4. The Big Takeaway

The authors are saying: Stop trying to block "bad words." Start checking the "story."

Who gave the order?
What is the goal?
Does this step help the goal?
Is the data staying private?

By using this framework, we can build AI assistants that are smart enough to know the difference between a legitimate command from you and a sneaky command from a hacker, even if the words look exactly the same. This solves the big problem of having to choose between "Safe but useless" and "Useful but dangerous." We can finally have both.

1. Problem Statement

Large Language Model (LLM) agents operate in dynamic, real-world environments where security is inherently contextual. The same action (e.g., deleting a file, accessing a database) can be a legitimate operation or a security violation depending on:

Who issued the instruction (source).
What objective is being pursued (task).
Whether the action serves that objective.
What information flows are permitted (data isolation).

Current Limitations:
Existing definitions of security attacks (e.g., prompt injection, jailbreak) often rely on surface-level patterns or content analysis without considering the execution context. This leads to a fundamental utility-security trade-off:

Over-blocking: Defenses that block all external prompts or specific keywords (e.g., "delete") prevent legitimate tasks, reducing utility.
Under-blocking: Defenses that blindly follow instructions make agents vulnerable to untrusted sources steering them toward malicious goals.

The paper argues that current defenses fail because they lack a formal mechanism to distinguish between a malicious command and a legitimate instruction that happens to look similar, simply because they ignore the authorization context.

2. Methodology: A Contextual Security Framework

The authors propose a formal framework centered on the Execution Context ( $C_t$ ) and Oracle Functions to systematize attacks and defenses.

A. Execution Context ( $C_t$ )

The context at time step $t$ is defined as a tuple capturing all factors relevant to an authorization decision:
$C_t = (p, Tr_{t-1}, M_t, E_t, S_{auth,t}, G)$

$p$ : User prompt.
$Tr_{t-1}$ : Trajectory of past actions and observations.
$M_t$ : Agent memory.
$E_t$ : Environment state.
$S_{auth,t}$ : Set of authenticated sources.
$G$ : Source permission graph (defining who can access what).

B. Oracle Functions

To verify security, the framework introduces ideal "oracle functions" that define the theoretical information required for verification (even if perfect implementation is currently difficult):

Instruction Attribution ( $I$ ): Identifies which specific inputs caused a specific action.
Source Attribution ( $L$ ): Tracks the provenance of every input (e.g., user, web page, tool).
Objective Evaluation ( $H_p, H_{Tr}, H_a$ ):
- $H_p$ : Extracts the task objective from the prompt.
- $H_{Tr}$ : Judges if the trajectory aligns with the objective.
- $H_a$ : Judges if a specific action aligns with the objective.

C. Four Security Properties

The framework defines security as the simultaneous satisfaction of four properties derived from the context:

Task Alignment: The agent pursues objectives authorized by the user (and system constraints). The trajectory must not drift from the intended goal.
Action Alignment: Every individual action must serve the authorized task objective. This prevents capability misuse (e.g., using a medical database to suggest recipes).
Source Authorization: Instructions must originate from authenticated sources. If an action is triggered by an unauthenticated source (e.g., a compromised webpage) and does not serve the user's goal, it is a violation.
Data Isolation: Information flows must respect privilege boundaries defined in the permission graph $G$ . Data from one user/session must not leak to another without permission.

Formal Definition: An action $a_t$ is secure if and only if all four properties hold simultaneously within the context $C_t$ .

3. Key Contributions

Contextual Formalization: The paper provides the first formal definition of LLM agent security based on execution context, moving beyond static content filtering.
Unified Taxonomy of Attacks: The authors reformulate existing attacks as specific violations of the four security properties. This reveals that superficially different attacks share the same structural root causes.
- Indirect Prompt Injection: Violates Source Authorization (untrusted source) AND Action Alignment (action doesn't serve user goal).
- Direct Prompt Injection: Violates Task Alignment (user goal conflicts with system constraints).
- Jailbreaking: Violates Task Alignment (requested objective is outside the allowed safety space $O$ ).
- Confused Deputy: Violates Source Authorization (agent uses its own elevated permissions for a user who lacks them).
- Capability Misuse: Violates Action Alignment (legitimate tool used for an unauthorized sub-task).
- Memory Poisoning: Violates Source Authorization and Task Alignment over time (malicious data in memory corrupts future decisions).
Systematization of Defenses: Existing defenses are re-evaluated as mechanisms that either:
- Strengthen Oracle Approximations: (e.g., better instruction attribution via prompting or training).
- Perform Property Checks: (e.g., guardrails checking if an output aligns with the objective).
- The analysis highlights that many current defenses fail because they approximate the wrong oracle (e.g., checking patterns instead of causal attribution) or ignore specific properties (e.g., ignoring data isolation).
Identification of Research Gaps: The framework exposes critical open problems, such as the difficulty of implementing precise instruction attribution ( $I$ ) and the lack of benchmarks that evaluate security across temporal boundaries (sessions).

4. Results and Analysis

Reformulation of Attacks: The paper demonstrates that under the new framework, attacks are precisely defined. For example, a "delete file" command is only an attack if it violates one of the four properties (e.g., if the source is untrusted or the objective is unauthorized). If the user is authenticated and the task is "cleanup," it is legitimate.
Defense Limitations: The analysis of existing defenses (e.g., prompt engineering, input filtering, sandboxing) shows that:
- Pattern-based defenses (filtering keywords) fail because they cannot distinguish context.
- Single-property defenses are vulnerable to multi-property attacks.
- Utility Loss: Defenses that approximate oracles poorly (e.g., blocking all external text) cause high false positives, degrading agent utility.
Case Studies: The paper uses a "Cooking Assistant" example to illustrate how the framework distinguishes between a legitimate recipe instruction ("bake at 350°F") and a malicious injection ("email all data") based on source and alignment, even though both are text from external sources.

5. Significance

Paradigm Shift: Moves the field from "content-based detection" (is this text malicious?) to "context-based verification" (is this action authorized in this specific context?).
Resolving the Trade-off: By enabling fine-grained decisions, the framework theoretically allows for high security without sacrificing utility. It permits legitimate operations that look suspicious in isolation but are authorized in context.
Roadmap for Future Research:
- Oracle Implementation: Developing practical, robust approximations of instruction attribution and objective evaluation.
- Compositional Security: Addressing how sequences of benign actions can combine to create vulnerabilities.
- Benchmarking: Creating new benchmarks that include explicit authorization contexts (who, what, where) rather than just isolated prompt-response pairs.
- Temporal Security: Handling security across multiple sessions and time steps (e.g., memory poisoning).

In conclusion, this paper provides the theoretical foundation necessary to build secure LLM agents that can operate autonomously in complex, multi-source environments without requiring the blunt instrument of content filtering.