Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-CodeX

1. Problem Statement

As AI agent systems evolve into diverse execution environments (e.g., tool-based agents, code-generation agents), existing safety benchmarks often fail to capture the specific risks inherent to these new settings.

• The Challenge: Agent frameworks remain architecturally stable, but their concrete execution settings, tool ecosystems, and interaction surfaces change rapidly. Static benchmarks cannot keep pace with these shifts, leading to gaps in safety evaluation.
• The Gap: There is a need for benchmarks that can evolve with agent settings without requiring a complete redesign of the evaluation infrastructure. Specifically, the paper addresses safety in two distinct, high-stakes environments:
  1. OpenClaw: An agent setting focused on tools, skills, sessions, and external actions (e.g., sending messages, file deletion).
  2. CodeX (OpenAI Codex/Codex-runtime): An agent setting focused on repositories, shell commands, patches, dependencies, and runtime policy boundaries.

2. Methodology

The authors propose a taxonomy-guided customization framework built upon the existing ATBench (Agent Trajectory Benchmark) infrastructure. Instead of building new benchmarks from scratch, they extend the ATBench "backbone" by adapting its core components to new domains.

Core Framework: ATBench

ATBench consists of two main pillars:

1. Unified Three-Dimensional Safety Taxonomy: A structured label space decomposing risk into:
   • Risk Source: Where the risk originates (e.g., User Input, External Entities, Internal Logic).
   • Failure Mode: How the agent fails (e.g., Malicious Tool Execution, Flawed Planning, Approval Bypass).
   • Real-world Harm: The consequence (e.g., Privacy Harm, Financial Harm, System Integrity).
2. Data Generation Engine: A pipeline that translates risk specifications into synthetic but realistic agent trajectories using taxonomy-guided sampling, heterogeneous tool sourcing, and planner-based synthesis.

Extension Strategy: Taxonomy Customization

The paper demonstrates that the ATBench engine remains static while the Safety Taxonomy is customized for each new setting. This allows the benchmark to evolve without redesigning the data generation pipeline.

• ATBench-Claw (OpenClaw Extension):

  • Focus: Stateful execution, session management, and cross-tool coordination.
  • Customization: Introduces new categories to capture execution-state risks not present in the baseline, such as Sender/Session Identity Ambiguity, Persistent Memory Contamination, Skill/Plugin Supply-Chain Compromise, and Cross-Channel Misrouting.
  • Schema Emphasis: Prioritizes tool snapshots, session states, and execution-action metadata.

• ATBench-CodeX (Codex Extension):

  • Focus: Repository manipulation, shell execution, and dependency management.
  • Customization: Uses a mixed strategy. It adds a few new categories (e.g., Repository Artifact Injection, Destructive Workspace Mutation, Unsafe Shell Execution) but primarily strengthens inherited categories with repository-specific interpretations (e.g., reinterpreting "Prompt Injection" as "Repository Artifact Injection" and "Tool Feedback" as "Build/Test Output").
  • Schema Emphasis: Prioritizes repository context, tool catalogs, and rollout traces over runtime events.

3. Key Contributions

1. ATBench-Claw and ATBench-CodeX: Two domain-specific benchmarks that extend the ATBench framework to OpenClaw and Codex environments, respectively.
2. Extensibility Framework: A novel methodology showing that agent safety benchmarks can be adapted to new execution settings by customizing the Safety Taxonomy rather than rebuilding the data generation engine. This ensures comparability across different agent architectures.
3. Granular Risk Coverage: The benchmarks provide fine-grained diagnosis capabilities, breaking down safety failures into specific risk sources, failure modes, and harm types relevant to code execution and tool-based agents.
4. Public Datasets: Release of large-scale, realistic trajectory datasets for both domains (hosted on Hugging Face) to facilitate future research.

4. Experimental Results

The authors evaluated various models (Guard models, Open-Source Instruct models, and their own AgentDoG system) on both benchmarks.

• Performance Trends:
  • ATBench-Claw: Models generally performed well. AgentDoG-Qwen3-4B achieved the best results (Accuracy: 0.872, F1: 0.896, Recall: 0.929). Specialized guard models (e.g., Llama-Guard) showed moderate performance, while large instruct models (Qwen3.5-397B) performed strongly.
  • ATBench-CodeX: Performance dropped significantly across the board compared to Claw, particularly for specialized guard models. This indicates that repository-centered execution and runtime policy constraints are harder to evaluate. AgentDoG-Qwen3-4B again led the pack (Accuracy: 0.822, F1: 0.838, Recall: 0.920).
• Difficulty Analysis:
  • Claw: Difficulties were concentrated in user-driven risks (jailbreaks) and prompt injection.
  • CodeX: Showed a "heavier long tail" of difficulty. Models struggled significantly with dependency/MCP supply-chain compromises, repository artifact injection, destructive workspace mutations, and unsafe shell execution.
  • AgentDoG Advantage: The AgentDoG system maintained robust performance across both benchmarks, particularly in the "long-tail" failure modes where other models collapsed, demonstrating the effectiveness of the guardrail framework.

5. Significance

• Scalable Safety Evaluation: The paper proves that safety benchmarks do not need to be reinvented for every new agent framework. By decoupling the generation engine from the domain-specific taxonomy, researchers can rapidly deploy safety evaluations for emerging agent settings.
• Realistic Risk Modeling: By focusing on execution contexts (sessions, repositories, shells) rather than just conversational content, the benchmarks address the most critical failure modes in real-world agent deployments (e.g., code injection, supply chain attacks).
• Diagnostic Granularity: The three-dimensional taxonomy allows developers to not just know if an agent is unsafe, but why (e.g., was it a failure in planning, a tool misuse, or a supply chain issue?), enabling targeted improvements in agent safety.
• Foundation for Future Work: The release of ATBench-Claw and ATBench-CodeX provides a standardized, high-quality foundation for evaluating the next generation of autonomous agents in complex, tool-rich, and code-centric environments.