Technical Summary: Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

Problem Statement

Current safety benchmarks for Large Language Model (LLM) agents often rely on static, pre-collected attack pools (fixed sets of prompts or jailbreaks) evaluated against defenders in single-turn or fixed multi-turn interactions. This approach fails to capture the dynamic nature of real-world adversarial threats, where an attacker can observe a defender's responses, adapt their strategy across multiple turns, and pivot when initial attempts fail. Furthermore, existing benchmarks typically evaluate a single attacker model against a defender, potentially missing vulnerabilities that only specific attacker architectures or capabilities can uncover. The authors argue that static pools freeze the adversary before measurement, leading to an incomplete understanding of agent security, particularly regarding indirect prompt injection and multi-turn jailbreaking.

Methodology

The paper introduces a new benchmark built on the AgentBeats multi-agent evaluation platform, designed to operationalize adaptive, multi-round attacks against memoryless defenders.

Core Design Principles

1. Multi-Turn, Agent-vs-Agent Battles: Each evaluation scenario consists of a 15-round dialogue. The attacker is an autonomous LLM agent that observes the defender's prior responses and conditions its next turn on that history.
2. Memoryless Defender: The defender is evaluated as a "fresh" interaction in every round, receiving only the current formatted input without access to prior conversation history. This isolates per-input robustness under sustained adaptive pressure, simulating an adversary who resets the session to avoid suspicion accumulation while the attacker learns across attempts.
3. Per-Battle Attack Regeneration: Unlike benchmarks using fixed attack pools, the attacker LLM generates each turn live during the battle. This ensures attacks adapt to specific defender behaviors rather than relying on pre-computed static sets.
4. Structured-Output Scoring: Defenders output structured JSON. Success is determined by a check_success function that verifies specific, scenario-relevant fields (e.g., a leaked password in a code field, an approved unsafe action in a decision field). This avoids ambiguity in free-form prose and prevents "reward hacking" where an attacker satisfies a proxy metric (like echoing a keyword) without achieving the actual threat.

Benchmark Construction

• Scenario Set: The benchmark comprises 21 held-out evaluation scenarios covering five threat classes: indirect injection (11), PII/data leak (6), insecure output (2), supply chain (1), and prompt extraction (1).
• Calibration: Scenarios underwent a three-stage calibration process (concept generation, plugin implementation, and quantitative filtering) using strong (GPT-5.2) and weak (GPT-4o-mini) models to ensure scenarios were discriminative (neither trivially easy nor impossibly hard) and required multi-turn interaction.
• Evaluation Matrix: A controlled 3×3 frontier evaluation was conducted using three top-tier attacker models (Claude Opus 4.6, GPT-5.4, Gemini 2.5 Pro) and the same three as defenders, resulting in 945 battles (N=5 per cell).
• Open Competition: An open competition on the gpt-oss-20b backbone involved 21 attacker and 20 defender teams, generating 18,422 battles on held-out scenarios to validate that the protocol captures strategy discovery beyond baseline sampling noise.

Key Results

1. Adaptive Attacks Reveal Significant Vulnerabilities

Restricting evaluation to the first attacker turn yields an Attack Success Rate (ASR) of 0–1%. However, allowing 15 rounds of adaptive attack increases the ASR to 5.4–14.0%. This demonstrates that multi-turn adaptation is the primary driver of success, not merely repeated independent sampling. Ablation studies confirm that removing the attacker's access to prior defender responses drops ASR to 0% on high-risk scenarios.

2. Model-Specific Weaknesses vs. Aggregate Rankings

While Claude Opus 4.6 and GPT-5.4 tied in aggregate ASR (5.4% each), their failure modes were sharply distinct:

• Opus was uniquely vulnerable in the memleak scenario (60% ASR vs. 7% for others), failing when a "regex-parser-debugging" pretext bypassed its refusal to share credentials.
• GPT-5.4 and Gemini held firm in memleak but Gemini was uniquely vulnerable in paperaccept (53% ASR vs. 0% for others), succumbing to a fabricated "Official Area Chair Note" that overrode its scoring rules.
• Gemini had the highest aggregate ASR (14.0%), driven by these specific scenario weaknesses.

3. Multi-Attacker Evaluation Uncovers Unique Attacks

Pooling three frontier attacker models uncovered 1.4–2.2× more unique successful attacks than the best single attacker alone. The generated attacks showed low cosine similarity (0.02–0.14) to attacks in existing benchmarks, indicating they represent novel strategies rather than paraphrases of known attacks. No single attack trajectory succeeded against all three frontier defenders, highlighting the heterogeneity of vulnerabilities.

4. Low Concordance in Defender Rankings

Using Kendall's W to measure agreement in defender rankings across scenarios, the benchmark yielded W = 0.19 (95% CI [0.02, 0.58]). This is significantly lower than benchmarks like b3 (W = 0.712), indicating that frontier defender weaknesses do not align on a single "safety" axis. A model that is robust in one scenario may be fragile in another, and aggregate ASR can obscure these critical, scenario-specific gaps.

Significance and Claims

The paper claims that static attack pools are insufficient for evaluating agent security because they freeze the adversary before the defender is measured. By operationalizing adaptive evaluation as a repeatable protocol, the benchmark:

• Surfaces Hidden Vulnerabilities: It reveals that aggregate safety scores can mask severe, model-specific weaknesses that only appear under specific multi-turn, adaptive conditions.
• Validates the Necessity of Multi-Attacker Pools: It demonstrates that relying on a single attacker model underestimates the attack surface, as different models discover distinct, non-overlapping failure modes.
• Provides a Repeatable Framework: The release includes the orchestrator, 21 scenarios, baseline harnesses, and a multi-attacker CLI, allowing the community to evaluate new defenders against live, adaptive attacks rather than static lists.
• Highlights the Limits of Single-Dimension Safety: The low concordance (Kendall's W) suggests that safety is not a single scalar property; a "safest" model may not exist across all threat vectors, necessitating scenario-specific defenses.

The authors explicitly state that their findings are based on the specific calibrated scenario set and that while they provide a robust protocol, the results do not constitute a definitive leaderboard ranking of models but rather a demonstration of how adaptive evaluation changes the measurement object. They release the data and code to support extensions into stateful defenders, broader attacker pools, and additional threat models.