Technical Summary: AgentSecBench

Problem Statement

Large Language Model (LLM) agents integrate trusted instructions, retrieved records, and tool observations into a single generative channel. This architectural choice conflates data flow with authority: untrusted strings (e.g., retrieved documents or tool outputs) can influence secret-bearing responses or action proposals even when no application policy authorizes such influence. Unlike conventional programs where type systems and access checks enforce the distinction between instruction and data, LLM agents often rely on prose within the system prompt to define these boundaries. The paper identifies a critical gap: while prompt text can describe a security boundary, it does not inherently enforce one. Consequently, an adversary who controls untrusted observations can bypass policy constraints, leading to prompt injection, privacy leakage, and unauthorized tool usage.

Methodology

The paper introduces AgentSecBench, an empirical instantiation of a formal security framework designed to measure and distinguish between prompt annotation and enforced security boundaries.

Formal Framework

The authors define intent-to-execution noninterference with permitted leakage. The core concept involves three "games" that share a cryptographic structure: an adversary selects an untrusted observation, a defense transforms the observable channel, and a distinguisher evaluates an unauthorized output event.

1. Instruction-Integrity Game: Tests if an adversarial instruction embedded in a document can override a benign user request.
2. Retrieval-Confidentiality Game: Tests if a secret (canary) assigned to a tenant outside the caller's authorization scope can be leaked via retrieved context.
3. Capability-Integrity Game: Tests if a tool observation can induce the selection of an action forbidden by the user's policy.

The framework distinguishes between policy projections (mathematical operations that remove unauthorized features from the input or action space) and prompt annotations (textual warnings or delimiters that leave the unauthorized data visible to the model).

Experimental Design

• Models: Experiments were conducted on Qwen3-0.6B and Qwen3-1.7B.
• Defenses: Six defense classes were evaluated:
  1. No defense (baseline).
  2. Delimiter hardening (prompt annotation).
  3. Classifier filtering (pre-generation pattern matching).
  4. Provenance-aware RAG gating (projection of retrieval context).
  5. Least-privilege tool gating (projection of tool outputs).
  6. Combined stack (composition of the above).
• Metrics:
  • Adversarial Advantage ($\widehat{Adv}$): The difference in violation rates between adversarial and paired benign controls.
  • Channel Closure ($\widehat{C}$): A binary indicator measuring whether the designated unauthorized marker or capability was removed from the input before generation.
  • Benign Utility: A lexical proxy for task completion on benign inputs.
• Protocol: The evaluation uses exact-match distinguishers (e.g., specific canary strings like BETA CANARY 0000 or action tokens like CALL SEND EXTERNAL EMAIL 0000) to ensure unambiguous ground truth. The benchmark avoids LLM judges to prevent introducing additional model-dependent variance.

Key Contributions

1. Formal Definition: Defined intent-to-execution noninterference with permitted leakage for agents combining trusted instructions with untrusted observations.
2. Security Distinction: Explicitly separated text-level instructions (prompt annotations) from enforced security boundaries (provenance projection and capability restriction). The paper argues that annotation relies on model compliance, whereas enforcement removes the adversarial channel.
3. Three Security Games: Instantiated instruction, secret, and capability isolation as policy projections, providing estimators for adversarial advantage, RAG leakage, and channel closure.
4. Evaluation Trace: Implemented an evaluation system that records both outcome events and the specific enforcement class that produced the defended context, allowing for the disentanglement of behavioral resistance from actual channel closure.
5. Empirical Analysis: Connected observed residual risk to whether an unauthorized symbol or capability was removed before generation, demonstrating that risk reduction does not always imply channel closure.

Results

The evaluation of 576 executions (across 3 games, 6 defenses, 2 models) yielded the following insights:

• Channel Closure vs. Advantage: The Combined defense achieved 100% channel closure and 0% adversarial advantage. In contrast, Delimiter hardening (prompt annotation) achieved 0% channel closure but reduced the advantage to 0.438 (slightly higher than the baseline of 0.375).
• Defense Effectiveness:
  • Provenance gating successfully closed the retrieval channel for the confidentiality game (0% leakage) but failed to close the capability channel in the tool game.
  • Least-privilege gating closed the capability channel (0% violation) but did not affect retrieval leakage.
  • Filtering achieved partial closure (54.2%) and reduced advantage to 0.042.
• Interpretation: A defense can reduce the observed attack success rate (ASR) while leaving the adversarial channel open (relying on the model to ignore the input). Conversely, a defense that closes the channel (removes the unauthorized feature) provides a stronger, enforceable guarantee, though it may still require post-generation validation to prevent model invention of the forbidden target.

Significance and Claims

The paper claims that AgentSecBench provides a security-oriented evaluation method that shifts the focus from ranking prompt templates to verifying the enforcement of security boundaries.

• Annotation vs. Enforcement: The primary contribution is the demonstration that prompt text can describe a boundary, but only provenance projection, capability restriction, and output validation can enforce one.
• Measurable Closure: The framework introduces "channel closure" as a measurable variable. This allows researchers to distinguish between a defense that successfully removes an unauthorized observation (enforcement) and one that merely reduces the probability of the model acting on it (annotation).
• Modest Claims: The authors explicitly state that the exact-marker experiments are "one observable instantiation" and do not claim exhaustive semantic security. The use of exact markers (canaries) serves as a sufficient witness for policy violation but under-approximates semantic violations (e.g., paraphrased secrets). The results establish insecurity for the specific games tested; non-violations do not certify unrestricted noninterference.

In conclusion, the paper argues that secure agent design requires explicit observation projection and capability validation alongside natural-language instructions, rather than relying on instructions alone to define the security perimeter.