Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation

1. Problem Statement

The evaluation of Agentic Large Language Models (LLMs) faces a fundamental trade-off between ecological validity (using real-world services) and reproducibility (using controlled sandboxes).

• Existing Benchmarks: Most benchmarks rely on mocked/synthetic environments (lacking realism) or live services (introducing temporal instability and non-determinism).
• Evaluation Flaws: Current evaluation metrics often rely on "trace matching" (checking if the agent called the right tools in order) or "LLM-as-a-judge" (subjective scoring). These methods fail to capture whether the actual outcome was achieved or if the agent caused unintended side effects (e.g., deleting unrelated data).
• The Gap: There is a lack of a framework that allows agents to interact with real enterprise APIs via code execution while maintaining a deterministic, reproducible environment for rigorous state verification.

2. Methodology: The Agent-Diff Framework

Agent-Diff addresses these challenges through a novel architecture combining containerized API replicas with state-diff-based evaluation.

A. Environment: Containerized API Replicas

Instead of mocking APIs or connecting to live production services, Agent-Diff builds containerized replicas of enterprise services (Slack, Box, Linear, Google Calendar).

• Behavioral Equivalence: The replicas are designed to be behaviorally equivalent to the real APIs for the specific set of requests exercised by the benchmark ($R^*$). They accept the same inputs and return the same outputs, but internal side effects (like analytics or notifications) are invisible.
• State-Centric Modeling: Each service is modeled as a state machine backed by a PostgreSQL database. The state $S$ consists of typed relational tables (e.g., channels, messages, files).
• Isolation: Every benchmark run executes in a fresh, isolated schema within the database, ensuring no cross-contamination between runs and allowing for concurrent execution.

B. Evaluation Mechanism: State-Diff Contracts

The core innovation is the State-Diff Contract, which decouples the process (how the agent acted) from the outcome (what changed in the environment).

• Snapshotting: The system snapshots the database state before ($T_{before}$) and after ($T_{after}$) task execution.
• Diff Calculation: It computes the precise difference ($\Delta S$) consisting of:
  • Inserted: New entities added.
  • Deleted: Entities removed.
  • Modified: Entities updated (comparing primary keys).
• Verification: Task success is determined by checking if the observed $\Delta S$ matches the expected state changes defined in declarative assertions.
• Closed-World Invariant: The evaluation enforces a strict invariant: any change in the environment state must be explained by the task's assertions. If the agent modifies or deletes an unrelated resource (an unintended side effect), the task is marked as a failure. This prevents "partial credit" for messy execution.

C. Benchmark Design

• Scope: 224 tasks across 4 enterprise services (Box, Slack, Linear, Google Calendar).
• Task Dimensions: Tasks vary by horizon (1–24 API calls), operation profile (Search, Create, Read, Update, Delete), entity scope (single vs. multi-entity), and information availability (explicit IDs vs. implicit discovery).
• Interaction Model: Agents interact via code execution (generating Bash/Python scripts using curl and standard utilities) rather than structured tool calling (JSON schemas).

3. Key Contributions

1. Novel Benchmarking Framework: Agent-Diff is the first framework to combine real API interface structures with fully reproducible, containerized sandboxes for enterprise SaaS tasks.
2. State-Diff Evaluation Contract: A rigorous, deterministic evaluation metric that verifies outcome rather than trace, capable of detecting unintended side effects and hallucinations.
3. Comprehensive Dataset: A curated suite of 224 complex, multi-step workflows requiring code generation, planning, and state management.
4. Empirical Analysis of Documentation: A systematic study on how API documentation impacts agent performance, distinguishing between "familiar" interfaces and "novel" endpoints.

4. Experimental Results

The authors evaluated 9 state-of-the-art LLMs (including DeepSeek-v3.2, Devstral, Qwen, Claude, Llama, etc.) under three conditions: No-Docs, Relevant-Docs, and All-Docs.

A. Model Performance

• Top Performers: DeepSeek-v3.2 (88.1% score) and Devstral-2512 (86.0%) formed a clear top tier.
• Bottom Performers: Llama-4-Scout (38.0%) and Claude-Haiku-4.5 (49.3%) struggled significantly.
• Service Variance: Performance varied by service; Linear (issue tracking) was generally easiest, while Calendar (temporal reasoning) was hardest for many models.

B. Impact of Documentation

• Pass Rate vs. Score: Providing relevant API documentation significantly increased the Pass Rate (+7.0 percentage points), meaning agents completed tasks more often. However, the Assertion-Weighted Score (which penalizes side effects) showed only a marginal, non-significant improvement.
• Context Dilution: Providing all documentation (All-Docs) did not improve scores and sometimes degraded performance on complex tasks (e.g., Linear), suggesting irrelevant context competes for attention.
• Novelty Test: A natural experiment using the recently introduced Box Hubs API (post-training cutoff) showed that documentation provided massive gains (+19.0 points) for Hub-specific tasks, whereas it had little effect on standard file tasks. This proves documentation is critical for inference-time learning of new APIs.

C. Error and Recovery Analysis

• Error Types: The most common errors were Reasoning Failures (53.6%) and Parameter Errors (43.3%). Endpoint Selection errors were rare but fatal.
• Recovery Strategies:
  • Top Models: Favored adaptive strategies like lookup_correct_value (searching for IDs) and break_into_steps (decomposition).
  • Bottom Models: Relied heavily on retry_same (repeating failed actions), which was strongly correlated with failure.
  • Hallucination: Documentation significantly reduced hallucination rates (by ~21 percentage points).

5. Significance and Conclusion

Agent-Diff represents a paradigm shift in agent evaluation by moving away from "did the agent call the right tool?" to "did the agent actually change the world as intended?"

• Robustness: The state-diff approach ensures that agents cannot "cheat" by hallucinating success or causing collateral damage.
• Realism: By using containerized replicas of real enterprise APIs, the benchmark captures the complexity of real-world workflows (authentication, pagination, error handling) without the instability of live services.
• Implications for Development: The results suggest that while LLMs are improving at code generation, their ability to plan, recover from errors, and utilize documentation for novel APIs remains a key differentiator. The framework provides a standardized, reproducible testbed for advancing agentic systems in enterprise productivity settings.

Code and Data Availability: The framework, benchmarks, and evaluation scripts are open-sourced under the MIT license at https://github.com/agent-diff-bench/agent-diff.