AutomationBench

1. Problem Statement

Current AI benchmarks for software automation fail to capture the complexity of real-world business workflows. Existing benchmarks typically focus on:

• Single-domain tasks: Operating within one application (e.g., a specific website or OS).
• Fixed toolsets: Providing agents with a pre-defined list of tools rather than requiring discovery.
• Lack of policy adherence: Ignoring the need to follow organizational rules or business logic documents.

Real business automation requires cross-application orchestration (spanning CRMs, email, calendars, etc.), autonomous API discovery (finding the correct endpoints without prior knowledge), and strict policy adherence. Current frontier models struggle to coordinate these elements simultaneously, leading to a gap between research benchmarks and actual business needs.

2. Methodology

Dataset Construction

• Domains: The benchmark covers six critical business domains: Sales, Marketing, Operations, Support, Finance, and HR.
• Scale: It includes 600+ tasks (100 public per domain for research, 100+ private per domain for evaluation) drawn from real Zapier workflow patterns.
• Environment: A simulated world with 47 applications and approximately 500 API endpoints.
• Hardening Techniques: To prevent overfitting and ensure difficulty, tasks include:
  • Noise: Irrelevant data, decoy records, and misleading context.
  • Ambiguity: Unclear locations for required information.
  • Policy Constraints: Business rules that override default behaviors or intuitive decisions.
  • Subjectivity Handling: Agents must record subjective judgments (e.g., lead scoring) as structured state changes rather than free-text.

Evaluation Pipeline

• Input: Agents receive a single natural language instruction (non-interactive).
• Tools: Agents interact via two primary tools:
  1. Search: Performs keyword (BM25) searches across API schemas to discover relevant endpoints.
  2. Execute: Mimics curl/fetch requests to call APIs (method, URL, body).
• Process: The agent must iteratively discover endpoints, make sequential/interdependent calls, and navigate the simulated environment.
• Grading (End-State Only):
  • Metric: Success is determined solely by the final state of the simulated applications (e.g., "Was the lead created in the CRM?" "Was the email sent?").
  • Method: Programmatic assertions verify invariant-based checks on the world state. There is no partial credit; the workflow must be fully completed.
  • Efficiency: Costs (USD per task) and token usage are tracked, but the primary score is pass/fail based on correctness.

3. Key Contributions

• First Cross-App API Benchmark: AutomationBench is the first benchmark to rigorously evaluate agents on cross-application coordination combined with autonomous API discovery and policy adherence.
• Realistic Complexity: Unlike previous benchmarks that use fixed tool lists, AutomationBench forces agents to search through schemas to find the correct endpoints, mimicking the "glue" work required in actual business automation.
• Objective, Deterministic Scoring: By using programmatic assertions on end-states rather than LLM-as-a-judge or subjective evaluation, the benchmark eliminates grading bias and ensures reproducibility.
• Hardening Against Reward Hacking: The inclusion of negative assertions (e.g., preventing mass-emailing to all users) and strict policy rules prevents models from "gaming" the system with shotgun approaches.

4. Results

The benchmark reveals a significant capability gap in current state-of-the-art (SOTA) models:

• Low Success Rates: Even the best frontier models score below 10%.
  • Opus 4.7 (max): 9.9%
  • Gemini 3.1 Pro (high): 9.6%
  • GPT 5.4 (high): 7.6%
  • Haiku 4.5: 1.5%
• Diverse Failure Modes:
  • False Confidence: Models frequently declare success despite failing (72–91% of failures).
  • Discovery Issues: Agents struggle to persist in searching for data when generic searches fail, often assuming data resides in the wrong application (e.g., CRM vs. Sheets).
  • Incomplete Processing: When processing lists, agents often summarize results without verifying every item.
  • Instruction Adherence: Models frequently paraphrase or ignore specific content requirements.
• Toolset Sensitivity: Performance improves significantly when using simplified "Zapier" tool schemas (12.8–14.3% for Gemini) compared to raw API schemas, indicating that schema complexity is a major bottleneck.
• Cost vs. Performance: While some models (like Gemini) achieve higher accuracy at lower costs, all models remain inefficient in terms of token usage and step counts for these complex tasks.

5. Significance

• Benchmarking Reality: AutomationBench provides a realistic measure of where AI agents stand relative to the capabilities businesses actually need. It highlights that current models are not yet reliable for autonomous, multi-step business orchestration.
• Research Direction: The benchmark identifies specific gaps—particularly in multi-step reasoning, policy adherence, and tool discovery—directing future research efforts toward solving these specific bottlenecks.
• Practical Pathway: By focusing on REST APIs (the universal standard for integration) rather than computer use (which is computationally expensive), the benchmark outlines a practical, near-term path for high-volume business automation.
• Community Resource: The public dataset, toolsets, and evaluation harness are available to accelerate research in agentic workflows, moving the field beyond single-application tasks.

In summary, AutomationBench establishes a new, high-bar standard for AI agents, demonstrating that while models can handle simple tasks, they currently lack the robustness to autonomously orchestrate complex, cross-application business workflows with strict policy compliance.