🤖 AI

ACE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments

ACE-Bench is a lightweight, reproducible agent evaluation benchmark that utilizes a unified grid-based planning task with static JSON environments to provide fine-grained, controllable manipulation of task horizons and difficulty, thereby addressing the high overhead and unreliable scoring issues prevalent in existing benchmarks.

Original authors: Wang Yang, Chaoda Song, Xinpeng Li, Debargha Ganguly, Chuang Ma, Shouren Wang, Zhihao Dou, Yuli Zhou, Vipin Chaudhary, Xiaotian Han

Published 2026-04-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wang Yang, Chaoda Song, Xinpeng Li, Debargha Ganguly, Chuang Ma, Shouren Wang, Zhihao Dou, Yuli Zhou, Vipin Chaudhary, Xiaotian Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to test how good a new robot chef is at planning a massive, complex dinner party for 100 guests.

The Problem with Current Tests
Right now, the way we test these AI "chefs" (agents) is like making them cook in a real, chaotic kitchen. They have to walk to the fridge, open the oven, call the grocery store, and wait for the delivery truck.

The Waste of Time: The paper says that in current tests, the robot spends nearly 40% of its time just waiting for the kitchen to respond (opening doors, loading websites, waiting for servers). It's like a student taking a math test but spending half the time waiting for the teacher to hand out the paper.
The Unfair Mix: Also, the tests are unbalanced. Some tasks are easy (like "make a sandwich"), while others are impossible (like "plan a wedding for 500 people"). If a robot is great at sandwiches but bad at weddings, the final score gets messy. You can't tell if the robot is smart or just lucky because it got easy questions.

The Solution: ACE-Bench (The "Digital Simulation Kitchen")
The authors built a new test called ACE-Bench. Instead of a real kitchen, they built a perfect, instant digital simulation.

Think of it like a video game level where the rules are written in a simple book (a JSON file). The robot doesn't have to wait for anything; it just flips through the pages instantly. This removes the "waiting time," so we only measure how smart the robot actually is.

How the Test Works: The "Schedule Puzzle"
The core of the test is a giant scheduling puzzle, like filling out a school timetable or a wedding seating chart.

The Grid: Imagine a calendar with empty slots. Some slots are already filled (pre-filled), but some are hidden and need to be filled by the robot.
The Rules:
1. Local Rules: "This class can't be on Monday." (Easy to check).
2. Global Rules: "The total number of credits for the whole week must be exactly 30." (Hard to check because you have to look at the whole picture).

Two Magic Knobs for Control
The genius of ACE-Bench is that the researchers can turn two "knobs" to make the test exactly as hard or long as they want:

Knob 1: The "Hidden Slots" (Horizon)
- Analogy: Imagine a maze.
- If you turn the knob to 1, the robot only has to fill in one empty seat. It's a short, simple task.
- If you turn it to 17, the robot has to fill in 17 seats, and every choice it makes affects the next 16. It's a long, complex chain of reasoning. This tests how far the robot can "think ahead."
Knob 2: The "Decoy Budget" (Difficulty)
- Analogy: Imagine a game of "Find the Real Key" where someone puts fake keys that look exactly like the real one.
- If the knob is 0, there are no fake keys. The robot just picks the right one.
- If the knob is 10, the robot is given 10 fake keys (decoys). These fake keys look perfect for the local rules (e.g., "Yes, this class fits on Monday!"), but if you pick them, they break the global rules (e.g., "Oops, now the total credits are too high!").
- This forces the robot to be very careful and think deeply, rather than just guessing.

What They Found
They tested 13 different AI models (from tiny ones to giant ones) on this new test.

The Results: The test worked perfectly.
- Bigger Brains Win: The giant AI models solved the hard puzzles much better than the small ones.
- Harder Puzzles = Lower Scores: As they turned up the "Hidden Slots" and "Decoy" knobs, the scores dropped for everyone, exactly as expected.
- No More Waiting: Because it's a lightweight digital test, they could run thousands of experiments very quickly, even while training the AI.

In Summary
ACE-Bench is like a gym for AI brains. Instead of making the AI run a marathon in the rain (waiting for slow websites), they put it on a treadmill where the speed and incline can be adjusted perfectly. This lets scientists see exactly how strong the AI's reasoning muscles are, without the noise of slow computers or unfair test questions.

1. Problem Statement

The paper identifies two critical limitations in existing Large Language Model (LLM) agent benchmarks (e.g., WebArena, TAU2-Bench, TerminalBench):

High Environment Interaction Overhead: Existing benchmarks often rely on complex, dynamic environments (web simulators, LLM-as-User, sandboxed executors). Interaction with these environments consumes a significant portion of evaluation time (34%–41%), making large-scale or training-time validation prohibitively expensive and slow.
Imbalanced Task Horizon and Difficulty: Current benchmarks suffer from uncontrolled variance in task complexity.
- Horizon Imbalance: Task lengths vary wildly (e.g., 10 to 100+ steps), leading to skewed aggregate scores where short tasks dominate.
- Difficulty Imbalance: Difficulty varies systematically across domains (e.g., retail tasks are easier than telecom tasks), causing simple domain averaging to mask true model capabilities.

2. Methodology: ACE-Bench Framework

The authors propose ACE-Bench, a unified, grid-based planning benchmark designed to decouple task complexity from environmental overhead.

Core Task Formulation

Grid Structure: Each task is an $R \times C$ grid (e.g., a 5-day $\times$ 7-time-slot schedule) where cells must be filled with items from a domain-specific pool.
Hidden Slots ( $H$ ): A subset of cells are "hidden" and must be filled by the agent. The rest are pre-filled.
Constraints:
- Local Slot Constraints ( $C_s$ ): Attribute restrictions for a specific cell (e.g., price $\le$ 460).
- Global Constraints ( $G$ ): Constraints applying to the entire grid (e.g., total credits $\ge$ 85, total price $\le$ 10,895).
Candidate Pool: For each hidden slot, the agent is presented with candidates categorized as:
1. Truth: The unique correct answer satisfying all constraints.
2. Filtered Candidates: Violate local constraints (easily eliminated).
3. Decoy Candidates: Satisfy local constraints but violate global constraints if selected.

Two Orthogonal Control Axes

Scalable Horizons (Controlled by $H$ ): The number of hidden slots determines the task horizon.
- $H=1$ : Single independent decision.
- $H=17$ : Long-horizon reasoning chain where each decision constrains future options.
Controllable Difficulty (Controlled by $B$ ): The "Decoy Budget" determines the number of globally misleading candidates.
- $B=0$ : No decoys; task is pure local filtering.
- $B>0$ : Agents must perform extensive global reasoning to distinguish truth from decoys.

Lightweight Environment Design

Static JSON Files: All task data and tool responses are stored in static JSON files.
Tool Interface: Agents interact via a unified set of tools (e.g., Query_attribute, Set_slot, Check_constraints, Done).
No External Dependencies: Tool calls are resolved by querying local files, eliminating the need for running services, web simulators, or external APIs. This reduces setup overhead to near zero.

3. Key Contributions

Configurable Evaluation Axes: Introduces a principled framework where task horizon ( $H$ ) and difficulty ( $B$ ) are independently and precisely tunable, allowing for systematic stress-testing of agent reasoning.
Lightweight Architecture: By removing dynamic environment dependencies, ACE-Bench enables fast, reproducible, and cost-effective evaluation suitable for training-time validation.
Domain Consistency & Discriminability: The benchmark spans six diverse real-world domains (Course Scheduling, Meal Planning, PC Build, Shopping, Travel, Workforce) while maintaining consistent difficulty structures, ensuring fair cross-domain comparison.
Comprehensive Empirical Validation: Validated across 13 models (ranging from 0.8B to 397B parameters) covering both Dense and Mixture-of-Experts (MoE) architectures.

4. Experimental Results

The authors conducted extensive experiments to validate the benchmark's properties and evaluate state-of-the-art models.

Horizon and Difficulty Control:
- Horizon: Interaction steps scale linearly with the number of hidden slots ( $H$ ), confirming controllable task length.
- Difficulty: Scores decrease monotonically as the decoy budget ( $B$ ) increases, confirming that the benchmark effectively measures the ability to handle global constraint conflicts.
Model Discriminability:
- Performance scales consistently with model size across families (Qwen3.5 Dense, Qwen3.5 MoE, MiniMax).
- Top Performer: Qwen3.5-397B-A17B achieved the highest overall score (84.9%).
- Scaling Trends: Smaller models (e.g., Qwen3.5-0.8B) scored near 0%, while larger models showed significant gains, demonstrating the benchmark's ability to distinguish capability levels.
Domain Consistency: Models showed consistent performance across different domains (e.g., Qwen3.5-27B scored between 74.1% and 79.6% across all 6 domains), indicating low domain bias.
Robustness (Tool Failure Simulation): Simulating tool call failures (10% and 30% rejection rates) showed that performance degrades significantly, particularly on high-horizon/high-difficulty tasks, highlighting the benchmark's utility for testing agent resilience.

5. Significance

Efficiency: ACE-Bench solves the "evaluation bottleneck" by removing the heavy computational cost of environment interaction, making it feasible to run thousands of evaluations quickly.
Interpretability: Unlike black-box benchmarks where a low score is ambiguous, ACE-Bench allows researchers to pinpoint why a model failed (e.g., inability to handle long horizons vs. inability to manage global constraints).
Training Utility: The lightweight nature makes it suitable for use during the model training loop (RLHF or SFT) to validate agent reasoning improvements in real-time.
Standardization: Provides a unified, controllable standard for comparing agent capabilities across different model architectures and scales, moving beyond the "one-size-fits-all" limitations of current benchmarks.

Code Availability: The authors have made the code and benchmark available at https://github.com/uservan/AgentCE Bench.