Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming

Technical Summary: Domain-Conditioned Safety in Frontier Computer-Using Agents

Problem Statement
Recent literature on Computer-Using Agents (CUAs) reports high Prompt Injection Attack Success Rates (ASR), ranging from 42% to 98%. However, these figures often rely on retired model checkpoints (e.g., GPT-4o, Claude 3.5 Sonnet) or the most vulnerable models within a test panel. Furthermore, many high-ASR results are driven by Reinforcement Learning (RL)-optimized injection strings that are not publicly released. This creates a reproducibility gap: it is unclear whether the techniques described in these papers remain effective against current frontier models when reproduced as human-readable, hand-crafted templates. Additionally, it is unknown if safety hardening in one domain (e.g., web browsing) generalizes to other agent modalities (e.g., coding).

Methodology
The authors introduce CUA-HANDCRAFTED, a public adversarial benchmark designed to evaluate frontier CUAs using non-RL, hand-crafted attack templates.

• Benchmark Design: The suite consists of 793 episodes across 24 multi-step web tasks on 8 self-hosted sites (e.g., Banking, HR Portal, CRM). It utilizes 56 attack templates spanning 8 categories (e.g., unauthorized action, data exfiltration, goal hijacking) and 5 depth levels.
• Attack Definition: "Hand-crafted" is strictly defined as human-readable prose strings written based on the textual descriptions in recent red-teaming papers (RL-Hammer, WASP, TRAP, MUZZLE). These templates contain no reward signals or automated optimization loops.
• Evaluation Targets: The primary evaluation targets the latest frontier models: Claude Sonnet 4.6 and GPT-5.4.
• Metrics: The study uses AgentDojo-compatible metrics: Benign Utility (BU), Utility-under-Attack (UuA), and Attack Success Rate (ASR). A "canary" detection system (using RFC 2606 .test URLs, specific ID patterns, or syntactic markers) verifies if an agent executed an injected goal.
• Ablation & Cross-Domain: The study includes prompt ablation (removing safety instructions) to determine if resistance is prompt-level or weight-level. It also introduces SKILLBENCH, a sister benchmark for coding agents, to test if safety hardening transfers across modalities.
• RL Baseline: A Phase 10 experiment runs an adaptive random-suffix RL attacker (based on AutoInject) to establish an upper-bound ceiling for RL-based attacks within a constrained budget.

Key Results

1. Zero ASR on Frontier Browser Agents: Against Claude Sonnet 4.6 and GPT-5.4, hand-crafted multi-step attacks achieved 0/140 success (0% ASR). The 95% Clopper-Pearson upper bound is 2.60%. The only non-zero hand-crafted ASR occurred in single-step Denial-of-Service (DoS) attacks on legacy models (6.16% on Sonnet 4, 6.88% on GPT-4o).
2. Resistance is in the Weights: Prompt ablation experiments (removing safety guidance or adding "helpful" instructions) maintained 0% ASR. This indicates that the resistance to browser-domain injection is a property of the model weights, not the system prompt.
3. RL Optimization is the Gap: While hand-crafted templates failed, an RL-based attacker baseline (within a $25 budget) also failed to find a successful string (0/50). The authors argue that the high ASR in literature (42–98%) is attributable to RL-discovered, off-distribution phrasing rather than the attack categories themselves. Benchmarks that do not release optimized strings are not reproducible.
4. Domain-Conditioned Safety (The Coding Gap): The same model weights (Sonnet 4.6, GPT-5.4) that resisted browser injection at 0% were highly vulnerable in the SKILLBENCH coding-agent benchmark. Hand-crafted skill-injection attacks achieved 100% ASR on Sonnet 4.6 and 79% ASR on GPT-5.4. This demonstrates that safety hardening is surface-specific; the browser domain is heavily red-teamed and hardened, while the coding domain remains vulnerable.
5. Reproducibility Audit: An audit of six recent CUA red-teaming papers found that four paired retired target models with unreleased optimized strings, rendering their headline ASR numbers permanently unreproducible at full fidelity.

Significance and Claims
The paper argues that current literature's high ASR numbers are largely artifacts of RL-optimized text and specific model generations, rather than inherent vulnerabilities in the attack categories. The authors claim that:

• Frontier safety is domain-conditioned: A model can be "safe" in a browser context while being "unsafe" in a coding context. Safety claims must specify the surface (e.g., "browser injection" vs. "coding skill injection").
• Reproducibility requires artifacts: Reporting techniques without releasing the optimized injection strings (or the attacker model) makes published ASR numbers unreproducible.
• Hand-crafted benchmarks serve as a floor: CUA-HANDCRAFTED provides a baseline for human-discoverable failure modes. If a model resists hand-crafted attacks, it has closed the most accessible failure modes, though it may still be vulnerable to RL-optimized attacks.

The authors conclude that future CUA safety reporting should treat ASR as a vector across surfaces, explicitly name target checkpoints, and release optimized strings or attacker models to ensure reproducibility. They do not claim CUAs are unattackable, but rather that the "gap" between current safety and high literature ASR is defined by RL optimization and uneven domain coverage.