🤖 AI

DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows

This paper introduces DecisionBench, a comprehensive benchmark substrate for evaluating emergent delegation in long-horizon agentic workflows across diverse models and tasks, revealing that while task quality remains stable across awareness conditions, significant unrealized potential exists for improving routing fidelity and overall orchestration performance.

Original authors: Yuxuan Gao, Megan Wang, Yi Ling Yu, Zijian Carl Ma, Ao Qu

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxuan Gao, Megan Wang, Yi Ling Yu, Zijian Carl Ma, Ao Qu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy kitchen. You have a complex order to fill that requires chopping, grilling, baking, and plating. You could try to do everything yourself, but you're tired, and maybe you're not the best at baking.

DecisionBench is a new "kitchen simulator" designed to test how well a manager (an AI agent) decides to delegate parts of the job to other chefs (other AI models) instead of doing it all alone.

Here is how the paper breaks down, using simple analogies:

1. The Problem: The "Do-It-All" Trap

In the past, we only tested if a single chef could cook a meal. But in the real world, AI agents have to do long, complicated tasks that take hours. Sometimes, a smaller, cheaper, or more specialized chef could do a specific step (like chopping vegetables) faster and better than the main chef.

The big question is: Does the main chef know when to ask for help, and do they ask the right person?

2. The Setup: The "DecisionBench" Kitchen

The authors built a controlled kitchen to test this. They didn't just watch the chefs cook; they set up a specific rulebook:

The Tasks: They used three existing "menus" of difficult tasks (GAIA, τ-bench, BFCL) that require long chains of steps.
The Chefs: They picked 11 different AI models from 7 different companies (like OpenAI, Anthropic, Google) to act as the "peers" available for help.
The Tools: The main chef has two special tools:
1. call_model: A button to hand off a sub-task to another chef.
2. read_profile: A menu that tells the main chef what the other chefs are good at (e.g., "Chef A is great at math but bad at long texts").

3. The Experiment: How Much Info Do We Need?

The researchers wanted to see how the main chef performs under different levels of information. They tested five scenarios:

Blind: The chef has the button to call for help, but no idea who is good at what. They just guess.
Aware (Preloaded): The chef is given a printed "cheat sheet" (a profile card) describing every other chef's strengths and weaknesses before the shift starts.
Aware (On-Demand): The chef has no cheat sheet, but can ask for a specific chef's profile only when they need it during the task.
The Variations: They tried different types of cheat sheets: one written by a human expert, one generated by hard math/stats, and one written by two other AI judges.

4. The Big Surprises (The Findings)

Surprise #1: Knowing more doesn't necessarily make the meal tastier.
Even when the main chef had perfect profiles of everyone else, the final quality of the meal (the task success rate) was almost exactly the same as when they were "blind."

Analogy: It's like having a detailed map of the city but still getting stuck in traffic. The extra information didn't automatically make the trip faster or the destination better.

Surprise #2: How you get the info matters more than the info itself.
This was the biggest discovery.

The "Cheat Sheet" (Preloaded) failed: When the chef was forced to read a long list of profiles before starting, they didn't use the info well. Their delegation choices were actually worse than when they were blind.
The "On-Demand" tool worked: When the chef could ask for a specific chef's profile only when they were stuck on a specific step, they made twice as many correct choices about who to call.
Analogy: Imagine a student taking a test. If you hand them a 50-page textbook before the test, they might get overwhelmed and forget the key facts. But if they are allowed to look up a specific fact only when they get stuck on a question, they solve the problem much better. The delivery method (asking when needed) was the hero, not the content of the book.

Surprise #3: We are leaving a lot of potential on the table.
The researchers calculated a "Perfect Ceiling." This is the score the chef would get if they magically knew exactly which helper was best for every single step and called them immediately.

The Result: The current best methods are still 15% to 31% worse than this perfect ceiling.
Analogy: We are currently driving a car at 60 mph, but the engine is capable of 100 mph. We have a huge amount of "unrealized headroom" to improve how we delegate tasks in the future.

Surprise #4: Chefs love their own brand.
The main chefs tended to call for help from chefs made by the same company, even when a chef from a different company was better suited for the job.

Analogy: It's like a manager at a Ford factory only calling for help from other Ford mechanics, even if a Toyota mechanic is the best in town for that specific repair. This "brand bias" was a clear pattern in the data.

5. The Conclusion

The paper concludes that DecisionBench is a new, essential tool for measuring how well AI agents manage teams.

The key takeaway for the future is: Don't just dump a bunch of information on an AI at the start. Instead, give them the tools to look up information exactly when they need it. If we fix how we deliver information, we might unlock the massive potential (the 15–31% gap) that is currently sitting unused.

The authors have released all their "kitchen recipes," the "chefs," and the "scorecards" so other researchers can test their own new ways of managing AI teams.

Technical Summary: DecisionBench

Problem Statement

Large Language Model (LLM) agents are increasingly deployed for long-horizon tasks involving minutes to hours of tool use. A critical inefficiency in current agentic workflows is the tendency to solve sub-tasks locally using a single, often over-provisioned model, rather than delegating to smaller or differently-specialized peer models. While routing sub-tasks to peers is a potential Pareto-frontier shifter, practitioners currently lack a benchmark to measure emergent delegation—the ability of an agent to autonomously decide whether to delegate, to whom, and what information to provide about peers. Existing benchmarks (e.g., GAIA, $\tau$ -bench, BFCL) focus on single-agent capability, while multi-agent systems research often relies on hand-coded role assignments rather than emergent behavior. Furthermore, cost-aware routing benchmarks typically treat routing as an external learned policy rather than an intrinsic behavior of the orchestrator.

Methodology: The DecisionBench Substrate

The authors introduce DecisionBench, a fixed benchmark substrate designed to isolate and measure emergent delegation in long-horizon agentic workflows. The substrate is agnostic to how peer information is generated or delivered, allowing for the evaluation of diverse orchestration methods.

Core Components

Task Suite: A fixed set of long-horizon tasks drawn from three established benchmarks: GAIA (general tool-use QA), $\tau$ -bench (tool-agent-user dialogue), and BFCL multi-turn (function calling). These are split into a Stage-1 (profiling) set and a Stage-2 (evaluation) set via a deterministic 20/80 stratified sample.
Peer-Model Pool: A heterogeneous pool of 11 models from 7 vendor families (OpenAI, Anthropic, Google DeepMind, DeepSeek, Moonshot, Qwen, MiniMax), pinned to a specific freeze date (2026-04-29) to ensure reproducibility.
Delegation Interface: The orchestrator agent operates with a call_model(name, subtask, budget_usd) tool to delegate sub-tasks. An optional read_profile(model) tool allows the orchestrator to retrieve structured descriptions of peers.
Annotation Layer: A frozen 7-skill taxonomy (e.g., tool-call schema adherence, multi-turn state tracking, information retrieval) and a deterministic, rule-based step tagger. This layer assigns skills to trajectory steps without LLM judgment, enabling process-level evaluation of whether a delegation was sent to a peer well-suited for the specific sub-task skill.
Metric Suite: A multi-axis profile covering:
- Outcome Metrics: End-task quality, cost, and latency.
- Process Metrics: Delegation rate, routing fidelity-at- $k$ (how often the agent picks the top- $k$ best peers for a given skill), vendor self-preference, and a counterfactual-delegation ceiling.

Reference Interventions

To characterize the substrate, the authors instantiate five reference conditions:

Blind: No peer information; delegates based solely on priors.
Aware-C1/C2/C3: Preloaded peer descriptions constructed via (C1) curated rubrics, (C2) deterministic statistics from Stage-1 traces, or (C3) dual out-of-pool LLM-judge summaries. These are paired with on-demand read_profile access.
Aware-Tool-Only: Identical to C2 but with the preloaded description suppressed, isolating the effect of the delivery channel (on-demand tool vs. preloaded prompt).

Key Contributions

The Benchmark Substrate: A reusable framework comprising a fixed task suite, peer pool, delegation interface, annotation layer, and multi-axis metric suite specifically for emergent delegation.
Reference Intervention Suite: A set of baselines (C1, C2, C3, and the delivery-channel ablation) to seed comparisons for future orchestration methods.
Empirical Characterization: A comprehensive sweep across 11 models, 3 benchmarks, and 5 conditions ( $N=23,375$ task instances) revealing that process-level metrics detect orchestration signals that outcome-only metrics miss.
Artifact Release: The substrate, annotation layer, 33 profile cards, analysis pipeline, and 220 run archives are released to enable reproducible research.

Key Results

The experimental sweep yielded three primary findings:

Quality is Flat Across Conditions: Mean end-task quality is statistically indistinguishable across the four awareness conditions ( $|\beta| \le 0.010, p \ge 0.21$ ). Relying solely on quality metrics would completely miss the orchestration signal.
Delivery Channel Dominates Content: Routing fidelity varies significantly.
- On-demand tool access (aware-tool-only) more than doubles routing precision-at-1 compared to the blind baseline (14.2% $\to$ 29.5%) while maintaining quality parity and lower mean cost.
- Preloaded descriptions capture less than half of this gain (C2: 20.8%, C3: 15.5%, C1: 7.5%).
- This indicates that the mechanism of information delivery (retrievable tool vs. static prompt) is more critical than the specific content of the peer description.
Significant Unrealized Headroom: A counterfactual-delegation ceiling (assuming perfect single-step delegation to the Stage-1-best peer) places perfect performance 15–31 percentage points above measured performance on every suite. This locates a large gap for future orchestration methods to close.
Vendor Self-Preference: Agents exhibit a "LLM-as-judge" style bias, over-delegating to peers from their own vendor family at rates 1.5–3.7 $\times$ higher than chance.

Significance and Claims

The paper claims that DecisionBench shifts the evaluation paradigm from outcome-only (quality) to process-level analysis (fidelity, routing behavior). The primary significance lies in demonstrating that:

Process metrics are essential: An orchestration method can significantly improve routing fidelity and cost-efficiency without immediately translating to higher end-task quality in current setups.
Delivery mechanisms matter: The "how" of peer information (on-demand tools) is a stronger driver of delegation quality than the "what" (description content).
Substantial room for improvement: The gap between current performance and the counterfactual ceiling suggests that current orchestration methods are far from optimal, particularly for mid-capability agents.

The authors remain modest, noting that the fidelity gains have not yet translated to quality improvements, potentially due to low delegation rates, partial signal in peer descriptions, or attention crowding from preloaded prompts. They posit that closing this gap requires delegating more frequently, multi-step delegation, or richer mid-task feedback—scenarios DecisionBench is designed to evaluate.