🤖 AI

Scaffold Effects on GAIA: A Controlled Comparison

This pre-registered controlled study demonstrates that scaffold choice significantly impacts agent performance on GAIA benchmarks, revealing that measured capability scores are highly scaffold-conditional and that the anticipated reduction in scaffold sensitivity as models improve does not hold, with structured multi-agent designs often yielding substantial accuracy gains over standard ReAct approaches.

Original authors: Jason Starace

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jason Starace

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge how good a chef is at cooking a complex meal. You have three different kitchens (scaffolds) to test them in:

The Solo Chef (ReAct): The chef thinks, grabs an ingredient, cooks, thinks again, and grabs the next ingredient, all in one continuous stream.
The Kitchen Team (Planner-Actor-Rater): A manager writes a recipe, a cook executes the steps, and a critic tastes the dish after every step to make sure it's right before moving on.
The Blueprint & Builder (Planner-then-Executor): First, a planner writes a detailed, step-by-step blueprint with no tools. Then, a builder takes that blueprint and builds the dish.

This study asked a simple question: Does the kitchen you put the chef in change how good they seem to be?

The answer is a resounding yes. In fact, the "kitchen" matters just as much as the chef's actual talent.

The Big Discovery: The "Kitchen" Matters More Than You Think

The researchers tested five different "chefs" (AI models from Anthropic, Google, and OpenAI) on a set of difficult puzzles (called GAIA). They found that simply switching the kitchen setup could change a model's score by 28 percentage points.

To put that in perspective: If you tested a model in a "Solo Chef" kitchen and it got a 56% score, but then you put the exact same model in a "Kitchen Team" setup, it might suddenly get an 84% score. The chef didn't change; the way they were allowed to work did.

This means that when we see headlines like "Model X is 80% smart," that number isn't just about the model. It's a mix of the model's brain plus the specific instructions and tools it was given. If you compare two models tested in different kitchens, you aren't comparing their brains; you're comparing their kitchens.

Myth Busting: "Smarter" Models Don't Need Less Help

The researchers had a hunch (hypothesis) that the most powerful, "frontier" models would be so smart that they wouldn't care much about the kitchen setup. They thought a super-brain could handle a messy kitchen just as well as a clean one.

They were wrong.

In fact, the most powerful model they tested (Anthropic's Opus) actually gained the most from having a structured, organized kitchen (the "Kitchen Team" setup) when the tasks were hard. The "smartest" model wasn't the most independent; it was the one that benefited most from having a manager and a critic. The gap between the best and worst performance didn't shrink for the smart models; it stayed wide or even got bigger.

The "Family" vs. "Rank" Surprise

Another surprise was that the benefit of a fancy kitchen depended on which family the model came from, not just how "ranked" it was.

The Anthropic family (Haiku, Sonnet, Opus) all got a huge boost from the "Kitchen Team" setup.
The models from Google and OpenAI didn't get that same boost.

It's like saying that a specific type of car engine runs much better with a specific brand of fuel, but a different brand of engine doesn't care. You can't just assume a "higher tier" model will always benefit more from better tools; it depends on who made the engine.

Cost and Efficiency

The study also looked at the "receipt" (cost).

The "Solo Chef" (ReAct) was the most expensive and slowest. It made a lot of mistakes and had to try again often.
The "Kitchen Team" and "Blueprint" setups were faster, made fewer mistakes, and recovered better when things went wrong mid-task.
The Winner: A specific combination of Google's Gemini model with the "Blueprint" setup was the cheapest and most accurate option for the hardest tasks.

The Bottom Line

This paper tells us that capability numbers are conditional. You cannot say a model is "X% capable" in a vacuum. You have to say, "X% capable when using this specific set of instructions and tools."

If you want to know how good an AI really is, you can't just look at one score. You have to realize that the score is a snapshot of the model plus the scaffolding holding it up. If you change the scaffolding, the score changes, sometimes dramatically. The "gap" between what a model can do and what it actually does in a test is huge, and it doesn't necessarily get smaller just because the model gets smarter.

Technical Summary: Scaffold Effects on GAIA

Problem Statement

Published agent capability scores often conflate a model's intrinsic ability with the performance enabled by its "scaffold" (the agent loop, tool orchestration, and prompting strategy). While the "elicitation gap"—the difference between what a model can do and what its scaffold lets it do—is acknowledged in the literature (e.g., METR's time horizon framework), its magnitude has not been rigorously characterized under controlled conditions across a frontier lineup of models. This study addresses the lack of controlled comparison to determine how much scaffold choice shifts measured accuracy, whether this sensitivity scales with model capability, and how it interacts with task difficulty.

Methodology

The study executes a pre-registered controlled comparison using the GAIA benchmark (validation Levels 1 and 2), which requires multi-modality, web browsing, file reading, and multi-step reasoning.

Experimental Design: A $3 \times 5$ $3 \times 5$ factorial design testing three scaffolds across five models.
- Scaffolds:
  1. S1 (ReAct): The standard single-agent alternating reason-act loop (baseline).
  2. S2 (Planner-Actor-Rater / PAR): A multi-agent design where a Planner decomposes tasks, an Actor executes tools, and a Rater reviews progress.
  3. S3 (Planner-then-Executor): A two-phase approach where a model generates a written plan without tools, followed by a separate execution phase.
- Models: Five models from three providers:
  - Anthropic: Claude Opus 4.7, Sonnet 4.6, Haiku 4.5.
  - Google: Gemini 3.1 Pro Preview.
  - OpenAI: GPT-5.5.
- Conditions: Tasks and conditions were held fixed. Each cell was run with three attempts per question.
Implementation: Evaluations were conducted using Inspect AI. A local patch was applied to the Google provider to enable concurrent use of native search and function tools.
Analysis: The study employs per-cell bootstrap confidence intervals (95%), mixed-effects logistic regression to test scaffold-by-model interactions, and cost-per-correct analysis. Three data slices were analyzed:
1. Primary: All attempt units (errors counted as incorrect).
2. Robust: Excludes units flagged by a provider-side serialization bug affecting Anthropic models.
3. Intersection: Retains only sample IDs where every (model, scaffold) cell had at least one clean completion.

Key Contributions

Isolated Scaffold Effects: A controlled comparison isolating the impact of scaffold choice on GAIA accuracy, holding model and task fixed.
Quantified Elicitation Impact: Statistical claims regarding the magnitude of the elicitation gap, moving beyond point estimates to include confidence intervals and interaction effects.
Reproducibility: Public release of code, configuration files, per-attempt logs, and the pre-registered sample ID snapshot.

Results

1. Magnitude of Scaffold Gaps (H1 Supported)

Scaffold choice alone moves measured accuracy by as much as 28 percentage points within a single model (Opus, Level 2, robust slice). The pre-registered hypothesis that scaffold variation produces at least a 10-point gap for at least one model is supported, particularly for Haiku across all slices and for Opus/Sonnet at Level 2 under the robust slice.

2. Scaffold Sensitivity and Model Capability (H2 Rejected in Direction)

The hypothesis that more capable models would be less scaffold-sensitive is rejected.

Statistical Finding: A mixed-effects model confirms that scaffold effects vary significantly by model ( $p < 0.0001$ ).
Directional Finding: Contrary to the prediction, the most capable Anthropic model (Opus) gains the most from structured scaffolds (S2/S3) at the harder level (Level 2), showing the largest gap (0.28). The "tier-scaling" claim (that gaps shrink as models improve) holds only for Level 1 under the robust slice but fails at Level 2.

3. Multi-Agent Advantage (H3 Conditionally Supported)

The hypothesis that multi-agent scaffolds (S2) outperform ReAct (S1) at Level 2 but not Level 1 is conditionally supported.

Family vs. Tier: The advantage is observed within the Anthropic family (Haiku, Sonnet, Opus) at Level 2, with statistically significant improvements.
Cross-Provider: The advantage is not observed for cross-provider models (Gemini, GPT-5.5), where gaps are indistinguishable from zero. This suggests model family, rather than capability tier, is the conditioning variable.

4. Planner-Executor on File Tasks (H4 Falsified)

The hypothesis that Planner-then-Executor (S3) outperforms on file-reading tasks is falsified.

Only one cell (Opus, Level 1) showed a supporting difference-of-differences under the primary slice, and this disappeared under the robust slice.
The result is bounded by "floor" effects (Haiku's poor document handling) and "ceiling" effects (Gemini/GPT-5 solving file tasks reactively at 100%).

5. Behavioral Signatures and Cost

Efficiency: Structured scaffolds (S2, S3) make significantly fewer tool calls and emit fewer tokens than ReAct (S1).
Recovery: At Level 2, structured scaffolds recover from mid-trajectory errors more often than ReAct, despite making fewer calls.
Cost: Gemini with S3 is the cheapest cell at both levels and the most accurate at Level 2. The Anthropic family exhibits a wider cost-per-correct dispersion at Level 2, indicating scaffold choice is a significant budget decision for these models.

Significance and Claims

The paper concludes that single-scaffold capability numbers are scaffold-conditional estimates, not intrinsic model properties.

The Elicitation Gap is Large and Persistent: The gap is measurable under controlled conditions and is not guaranteed to shrink as models improve; in fact, it may grow for capable models on hard tasks.
Comparability Issues: Comparing two models evaluated under different scaffolds is "close to uninterpretable" as a model comparison, as the difference may be entirely attributable to the agent loop fit.
Procurement Implications: For evaluators with fixed budgets, choosing a poor scaffold results in measuring less capability while paying more per unit of measured capability.
Methodological Shift: The study argues that capability evaluations must move beyond single-scaffold point estimates to account for the interaction between model, scaffold, and task difficulty.

The authors remain modest about the scope, noting that the multi-agent advantage is specific to the Anthropic family and that the file-reading hypothesis requires a larger task pool and models positioned between floor and ceiling for a fair re-test. The study does not claim to have solved the elicitation problem but provides the first controlled quantification of its magnitude on a general-assistant benchmark.