Technical Summary: AgentRedBench and AgentRedGuard

1. Problem Statement

The paper addresses indirect prompt injection in Large Language Model (LLM) agents that operate via tool-use integrations (e.g., Gmail, Salesforce, Jira). Unlike traditional prompt injection where the user directly inputs malicious text, indirect injection occurs when an agent reads content from a third-party integration (a "read" operation) that an external adversary has poisoned. The agent then executes actions ("writes") based on this untrusted content, potentially leaking data or performing unauthorized operations.

The authors argue that existing benchmarks and defenses are insufficient for three reasons:

1. Limited Coverage: Existing benchmarks cover only a handful of integrations, whereas production agents interact with a vast pool of distinct SaaS tools.
2. Static Attack Content: Most benchmarks replay fixed attack payloads. They fail to capture dynamic attacker behavior where the payload is adapted per run based on the specific integration schema and feedback.
3. Training Data Mismatch: Open-source safety guards (e.g., Llama Guard, WildGuard) are trained on chat-style dialogue, not the specific distribution of tool-response content that agents actually consume. This creates a blind spot where guards fail to detect adversarial patterns embedded in API responses.

2. Methodology

AgentRedBench (The Benchmark)

The authors introduce AgentRedBench, a dynamic, LLM-driven redteaming benchmark designed to measure these threats.

• Scope: It covers 215 subtle "underspecified-authorization" scenarios across 24 enterprise integrations spanning nine functional families (e.g., communications, CRM, HRIS).
• Attack Taxonomy: The benchmark focuses on attacks that exploit the boundary of user authorization rather than overt "ignore previous instructions" commands. It defines five attack types:
  • Active Types: output_channel_url_relay (embedding attacker URLs in summaries), destination_hijack (substituting email recipients), and content_hijack (substituting message bodies).
  • Bound-Delegation Types: tool_argument_hijack and tool_family_creep (attempting to add arguments or call adjacent tools), which serve as baselines for what models successfully resist.
• Dynamic Generation: Unlike static benchmarks, the attack content is generated per-run by an attacker LLM. The attacker is conditioned on the integration schema, the specific attack type, and feedback from a judge on previous attempts. This ensures the payload is tailored to the specific tool response structure.
• Evaluation Pipeline: The system uses an orchestrator to inject poisoned content into mock integration states. A target agent attempts to complete a benign user task. An LLM judge scores the trace for success (attack executed), partial success, or failure.

AgentRedGuard (The Defense)

The authors propose AgentRedGuard, a defense mechanism specifically trained on the distribution of adversarial tool-response content.

• Architecture: Unlike LLM-based guards, AgentRedGuard is a fine-tuned text classifier (using a 23M parameter MiniLM model). The authors argue that detection at the integration layer relies on recognizing surface signatures (e.g., audit-log markers, specific domain patterns) rather than deep semantic reasoning, making a lightweight classifier more efficient and effective.
• Training Data: The model is trained on a corpus of adversarial tool responses extracted from the benchmark traces, paired with a "hard-negative" benign corpus derived from real enterprise tool outputs (HRIS, ATS, CRM).
• Generalization Strategy: To prevent memorization, the training set excludes four specific integrations (Slack, Linear, Salesforce, Calendar) and holds out specific attack types to test cross-integration and cross-attack-type generalization.

3. Key Results

Vulnerability of Unguarded Agents

Across an eight-model panel (Anthropic, OpenAI, Google), the Attack Success Rate (ASR) without any guard is severe:

• Range: ASR ranges from 32.1% (Claude Sonnet 4.6) to 81.4% (Gemini 3 Flash).
• Alignment vs. Scale: The paper notes that resistance is driven by alignment training rather than model scale. For instance, within Anthropic, Sonnet 4.6 is significantly more resistant (32.1%) than Haiku 4.5 (79.5%), despite Haiku being a smaller model.
• Baseline Failure: Existing open-source guards (Llama Guard 3, PromptGuard 2, WildGuard, ProtectAI) fail to detect these attacks effectively when applied to tool-response content. For example, PromptGuard 2 achieved 0.00% True Positive Rate (TPR) on the held-out test set, and Llama Guard 3 achieved only 0.15% TPR.

Effectiveness of AgentRedGuard

AgentRedGuard significantly mitigates the threat while maintaining low overhead:

• ASR Reduction: It reduces the overall panel ASR from 69.9% to 2.4%.
• Performance Metrics:
  • True Positive Rate (TPR): 99.75% on cross-integration held-out data.
  • False Positive Rate (FPR): 0.37% on a production-shaped benign corpus.
  • Latency: Adds a median of 9.5 ms per tool response on commodity CPUs, requiring no GPU.
• Generalization: The model maintains high performance on integrations it was never trained on (cross-integration) and on attack types excluded from its training set (cross-attack-type), confirming it learns generalizable surface markers rather than memorizing specific examples.

4. Significance and Claims

The paper claims that AgentRedBench and AgentRedGuard represent a necessary shift in how agent security is evaluated and defended:

1. Dynamic Evaluation: The work demonstrates that static benchmarks under-measure the threat. Dynamic, schema-conditioned attack generation reveals vulnerabilities that fixed payloads miss.
2. Distribution Shift: The primary failure mode of existing guards is a distribution shift; they are trained on chat data, not tool-response data. AgentRedGuard proves that a classifier trained specifically on the integration-layer distribution can detect these subtle attacks with near-perfect accuracy.
3. Practical Defense: By using a lightweight classifier rather than a large LLM, the defense is deployable with minimal latency and cost, making it viable for real-time interception in the tool-calling loop.
4. Open Science: The authors release the codebase, integration schemas, and the AgentRedGuard model weights openly. However, to preserve the benchmark's validity over time, the canonical scenario set is gated through a maintainer-mediated channel to prevent the scenarios from entering training corpora (which would render the ASR metrics meaningless due to memorization).

The paper concludes that while alignment training provides a baseline of resistance (as seen in Sonnet 4.6), inference-time defenses like AgentRedGuard are essential for the broader adoption of LLM agents in untrusted, multi-integration environments. The multi-integration chained attack surface remains an open frontier for future research.