🤖 AI

Anchor: Mitigating Artifact Drift in Agent Benchmark Generation

This paper introduces Anchor, a task-generation pipeline that mitigates artifact drift by formalizing business workflows into constraint optimization programs to produce auditable, verifiable, and scalable evaluation environments, demonstrated through the release of ERP-Bench, a benchmark of 300 long-horizon tasks in a production-grade ERP system.

Original authors: Maksim Ivanov, Abhijay Rana

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maksim Ivanov, Abhijay Rana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to run a factory. You want to give it a test to see if it's smart enough to handle real business tasks like ordering supplies, building products, and sending invoices.

The problem, according to this paper, is that most of these tests are broken. They suffer from what the authors call "Artifact Drift."

The Problem: The "Broken Recipe"

Think of creating a test for an AI agent like writing a recipe for a cake. To make a fair test, you need four things to match perfectly:

The Instructions: "Make a chocolate cake."
The Kitchen (Environment): The oven, bowls, and ingredients you give the robot.
The Answer Key (Oracle): The perfect cake you expect to see at the end.
The Judge (Verifier): The person who tastes the cake and decides if it's good.

In most current AI tests, these four things are written by different people or generated by different tools that don't talk to each other.

The Instructions might say, "Use 2 cups of sugar."
But the Kitchen only has 1 cup of sugar.
The Answer Key assumes the robot used 3 cups.
The Judge might give a passing grade even if the cake is burnt, just because it looks like a cake.

When these four parts don't agree, the test is unfair. The robot might fail because the test was impossible, or it might "cheat" (hack the reward) by finding a loophole the test creator didn't intend. The authors call this mess Artifact Drift.

The Solution: "Anchor"

The authors built a new system called Anchor to fix this.

Imagine instead of writing four separate documents, you write one single, perfect mathematical blueprint (a "constraint program"). This blueprint is like a master recipe that defines exactly how the cake should be made, what ingredients are available, and what the rules are.

When you want to create a test:

You tweak the blueprint slightly (e.g., "Make it harder by adding more customers" or "Make it easier by giving more stock").
A computer solver reads this blueprint and says, "Okay, here is the exact perfect way to solve this specific version."
Anchor then automatically translates that one perfect solution into all four parts of the test:
- It writes the Instructions in plain English.
- It sets up the Kitchen with the right ingredients.
- It creates the Answer Key based on the solver's math.
- It programs the Judge to check against those exact same rules.

Because everything comes from that single blueprint, they can never disagree. The instructions, the environment, the answer, and the judge are all perfectly aligned.

The Result: ERP-Bench

Using this system, the authors created a new test suite called ERP-Bench.

What is it? It's a set of 300 realistic business tasks involving buying parts, making products, and managing inventory, all inside a real-world business software system called Odoo.
Why is it special? It's "verifiable." The system knows the mathematically perfect answer for every single task. It can tell you exactly how close the AI got to the perfect solution, not just if it "looked" right.

What They Found

They tested five of the smartest AI models available today on these 300 tasks. Here is what happened:

The Difficulty Scale Works: They could make the tasks "Easy," "Medium," or "Hard" just by changing numbers in their blueprint. The AI models got worse as the tasks got harder, exactly as predicted.
The "Good Enough" vs. "Perfect" Gap: The AI models were okay at following the basic rules (like "don't run out of stock"). They got the basic constraints right about 26% of the time. However, they were terrible at finding the best possible solution (like spending the least amount of money). They only found the perfect, optimal solution 17% of the time.
The Interface Matters: The AI models did much better when they could "talk" to the software directly (like a programmer using code) compared to when they had to "click" buttons on a screen like a human (using a mouse or browser). The "clicking" methods were much slower and made more mistakes.

The Bottom Line

The paper argues that to build AI that can actually do real business work, we need to stop building tests with mismatched parts. By using a "single source of truth" (the Anchor system), we can create fair, auditable, and realistic tests that tell us exactly how good an AI agent really is at solving complex business problems.

Technical Summary: Anchor – Mitigating Artifact Drift in Agent Benchmark Generation

Problem Statement

Current benchmarks for enterprise AI agents suffer from a fundamental construction flaw termed artifact drift. In traditional benchmark generation, the four critical components of a task—the instruction ( $I$ ), environment ( $E$ ), oracle solution ( $x^*$ ), and verifier ( $V$ )—are often created through loosely coupled processes. This decoupling leads to inconsistencies where artifacts disagree on task requirements. For example, an environment may lack data assumed by the instruction, an oracle may depend on state omitted by the environment, or a verifier may accept outcomes not requested by the instruction. These inconsistencies result in environments that are unsolvable, susceptible to reward hacking, or simply inconsistent, creating a gap between benchmark scores and real-world production performance.

Methodology: The Anchor Pipeline

To address artifact drift, the authors introduce Anchor, a task-generation pipeline that compiles all four task artifacts from a single, solved specification. The pipeline operates as follows:

Formalization: A domain expert and an engineer formalize a business workflow (e.g., invoice prioritization, production scheduling) as a parametric constraint program using OR-Tools CP-SAT. This specification includes decision variables, business-rule constraints, and an objective metric.
Solving: For a given set of parameters ( $\theta$ ), the CP-SAT solver either rejects the instance as infeasible or certifies an optimal solution ( $x^*_\theta$ ). This result, combined with the parameters and constraints, forms a solved specification ( $S_\theta$ ).
Deterministic Compilation: Four translation layers deterministically project $S_\theta$ $S_{θ}$ into the task artifacts:
- Instruction Generator: Renders parameters, constraints, and objectives into natural language.
- Setup Generator: Writes sampled initial records into the environment container.
- Oracle Generator: Writes the solver's certified solution as the reference terminal state.
- Verifier Generator: Grades terminal states against the program's constraints and objective.

Because all artifacts are deterministic projections of the same solved specification, inconsistencies are mitigated by construction. The pipeline includes five end-to-end checks (no-op agent scoring, oracle replay, LLM consistency judging, reward-hacking canaries, and expert spot-checks) to surface residual defects.

Key Contributions

1. The Anchor Pipeline

A novel task creation framework that unifies instruction, environment, oracle, and verifier generation from a single constraint program. This approach ensures that difficulty is tunable via parameters and that optimal solutions are mathematically certified, enabling verifiable training curricula.

2. ERP-Bench

A benchmark of 300 long-horizon tasks spanning procurement and manufacturing workflows within Odoo 19, a production-grade open-source ERP system.

Scope: Covers 29 workflow patterns, including routine replenishment, disruption recovery, and production-and-fulfillment scenarios.
Complexity: Tasks involve structured data (sales orders, purchase orders, manufacturing orders, BOMs) and require agents to navigate complex business rules (lead times, MOQs, capacity constraints).
Verification: The verifier uses a weighted scoring system (25% constraint satisfaction, 60% optimality, 15% traceability) with hard-zero gates for critical failures.

3. Comprehensive Evaluation

The authors evaluated five frontier models (GPT-5.5, Claude Opus 4.7, GLM-5.1, GLM-5V-Turbo, Kimi K2.5) across three harnesses (coding, browser, computer-use) over 18,000 trials.

Harnesses: Built on the open-source pi-mono toolkit to reflect real-world scaffolds rather than benchmark-specific wrappers.
Metrics: Evaluated Pass@1, Pass@5, average scores, and the feasibility-optimality gap.

Results

1. Difficulty Correlation

The parametric difficulty intervention successfully predicts realized performance. Across all models and harnesses, Pass@5 scores drop monotonically from easy to hard tiers:

Coding: 70.5% $\to$ 22.3%
Browser: 46.5% $\to$ 7.7%
Computer-use: 56.0% $\to$ 9.5%

Task parameters such as the number of workcenters, manufacturing orders, and distinct vendors show strong negative correlations with Pass@1.

2. The Feasibility-Optimality Gap

A significant gap exists between satisfying constraints and achieving optimality. Across all models, agents satisfy explicit task constraints in 26.1% of trials but reach a fully optimal solution in only 17.4%. The strongest model (GPT-5.5 in coding) shows a 23.4-point gap. Score degradation is driven primarily by broken explicit constraints (feasibility) rather than suboptimal trade-offs among feasible plans.

3. Harness Performance

Models perform significantly better in the coding harness (API-driven) compared to browser or computer-use harnesses.

GUI harnesses resolve 16–56 percentage points fewer tasks than the coding harness.
GUI harnesses cost 3.1–14.3× more per attempted task due to higher turn counts (action batching limitations in UI vs. API).
No instances of reward hacking were observed during evaluation.

4. Reliability

Current frontier agents are not yet reliable enough for unattended deployment. While Pass@5 (at least one success in five trials) is measurable, Pass5 (all five trials succeeding) is roughly an order of magnitude lower (e.g., 9.8% in coding), indicating high variance in agent performance.

Significance and Claims

The paper claims that Anchor and ERP-Bench provide a concrete recipe for building auditable evaluation environments for economically valuable agent work. By formalizing business workflows into constraint optimization programs, the authors demonstrate that it is possible to generate realistic, verifiable, and scalable benchmarks that mitigate the artifact drift plaguing current evaluations.

The authors position this work as a step toward bridging the gap between benchmark scores and production performance. They note that while the framework prioritizes feasible, reproducible, and exactly gradable tasks, it is naturally suited for enterprise work visible in terminal system states (like ERP operations) but less suited for workflows relying on tacit judgment or negotiation. The release of the task generator and dataset at erpbench.ai is intended to support further research into verifiable agent training and evaluation.