Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents

Based on the provided text, here is a detailed technical summary of the paper "Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents."

Problem Statement

The paper addresses the vulnerability of tool-using Large Language Model (LLM) agents to indirect prompt injection. Unlike standard chatbots where an injection results in harmful text, agents with tool access (e.g., database queries, API calls, payments) face an authorization problem: an attacker can plant instructions in untrusted data (emails, web pages, PDFs) that the agent executes as legitimate actions.

The authors argue that the first generation of defenses (in-band), which rely on training models to recognize and refuse malicious text or using guardrail classifiers, has failed against adaptive attackers. Research by Nasr et al. and others demonstrated that static benchmarks are insufficient; when attackers optimize specifically against a defense, previously "secure" in-band defenses suffer attack success rates (ASV) exceeding 90%.

Consequently, the field has shifted to out-of-band defenses (second generation), which move enforcement outside the model using deterministic policy monitors (e.g., CaMeL, FIDES, Progent, FORGE). However, these new defenses are currently evaluated using the same flawed methodology: static benchmarks with fixed attack sets. The paper posits that without testing against adaptive, defense-aware attackers, confidence in these systems is premature.

Methodology and Framework

The paper employs a two-pronged approach: a theoretical systematization of existing defenses and an empirical adaptive evaluation.

1. Theoretical Systematization (Sections 2–6)
The authors organize out-of-band defenses through classical computer security primitives, arguing that these modern systems are instances of concepts established in the 1970s:

• Biba Integrity: Enforcing "no write up" rules where a subject reading untrusted (low-integrity) data cannot authorize high-integrity actions.
• Reference Monitors: A deterministic layer that mediates every access, must be tamper-proof, and enforce complete mediation.
• Saltzer–Schroeder Principles: Applying least privilege and fail-safe defaults.
• Capabilities and Information Flow Control (IFC): Using unforgeable tokens and taint tracking to propagate integrity labels.

The authors compare major systems (Dual-LLM, CaMeL, FIDES, Progent, RTBAS, FORGE) across eight dimensions, including enforcement primitive, determinism, monitor location, and coverage of integrity vs. confidentiality. They note that while these systems effectively gate actions, many struggle with implicit flows (side channels) and confidentiality.

2. Adaptive Evaluation Protocol (Sections 10–11)
The paper defines a protocol for adaptive evaluation to replace static benchmarks. The protocol requires:

• Targets: Open-source, reproducible action-level defenses (specifically Progent).
• Attacker Model: A defense-aware, adaptive attacker who knows the policy and can optimize adversarial strings, spoof provenance, and probe side channels.
• Metrics: Action-level Attack Success Value (ASV), False Negative Rate (FNR), utility retention, and monitor overhead.

3. Empirical Execution
The authors executed this protocol against Progent, an open-source out-of-band defense, on the AgentDojo benchmark.

• Setup: They used an open-weight agent (Qwen2.5-7B) self-hosted on a single H200 GPU, a configuration not tested in Progent's original paper.
• Attack Strategy: They utilized a hand-crafted adaptive attack and extended Progent's own adaptive-attack analysis from their Appendix E.
• Comparison: They compared the results against Progent's static benchmark performance.

Key Contributions

1. Structured Comparison: The paper provides the first unified framework comparing out-of-band defenses using classical security primitives (Biba, Reference Monitors, IFC), clarifying what each system covers and where they share structural limitations (e.g., side channels).
2. Identification of an Evaluation Gap: The authors highlight that the field lacks a standardized, independent, defense-aware adaptive evaluation for action-level defenses, warning that the community is repeating the evaluation mistakes made with in-band defenses.
3. Protocol Specification: They define a concrete protocol for adaptive evaluation that can be applied to any action-level defense.
4. Empirical Reproduction: They performed an independent reproduction and extension of Progent's adaptive analysis, testing it on a weaker, open-weight model (Qwen2.5-7B) rather than the proprietary models used in the original study.

Results

The empirical evaluation yielded the following results:

• Static Baseline: Progent reduced mean attack success on AgentDojo from 25.8% (undefended) to 4.2% in the authors' setup.
• Adaptive Attack: When subjected to a hand-crafted adaptive attack, the defense held. The attack success rate did not increase; it remained low at 2.6%.
• Comparison to Original: This result is consistent with Progent's original findings (which reported a drop from 39.9% to 1.0% on AgentDojo), suggesting the defense is robust against this specific adaptive attack vector on this specific model.

Limitations of Results:
The authors explicitly state these results are a "small-scale data point." The evaluation used:

• A single, weaker open-weight model (Qwen2.5-7B).
• A single black-box attack template.
• A hand-crafted adaptive attack rather than a fully optimized white-box attack (e.g., GCG).
• The paper notes that a stronger, optimized white-box attack remains an open question.

Significance and Claims

The paper makes a modest but critical claim: Deterministic out-of-band enforcement appears to be a harder target for adaptive attackers than in-band detection, but this hypothesis is not yet established.

• Consistency, Not Proof: The results are consistent with the hypothesis that moving enforcement outside the model improves security, but the paper does not claim to have proven the absolute robustness of these systems.
• The "Unknown": The primary significance is the identification of the evaluation gap. The field is currently "confident" based on static benchmarks, a state the authors argue is dangerous given the history of in-band defenses.
• Remaining Challenges: Even if the defenses hold against adaptive attacks, the paper identifies unresolved issues:
  • Provenance: The trusted base relies on correctly assigning provenance to user inputs, which is often under-specified.
  • In-the-Loop Tasks: Tasks requiring the model to act on untrusted content (e.g., "read this email and add to calendar") remain difficult to secure without reintroducing human judgment or causing approval fatigue.
  • Side Channels: Deterministic monitors may still be vulnerable to implicit flows and timing channels.
  • Text-to-Text Harms: These defenses protect actions but do not prevent injections that result in misleading text summaries or instructions to the user.

In conclusion, the paper argues that while out-of-band defenses represent a necessary architectural shift, the community must transition from static benchmarking to rigorous, adaptive evaluation to validate their true security posture.