🤖 AI

AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?

The paper introduces AgentFloor, a tiered benchmark demonstrating that small open-weight models can effectively handle the majority of routine, short-horizon tool-use tasks in agentic workflows, reserving large frontier models only for complex long-horizon planning while achieving comparable aggregate performance at a lower cost.

Original authors: Ranit Karmakar, Jayita Chatterjee

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ranit Karmakar, Jayita Chatterjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a busy delivery company. Every time a customer places an order, your system has to make dozens of tiny decisions: "Check the address," "Look up the price," "Call the warehouse," "Print the label."

For a long time, the rule was: "Use our most expensive, super-smart CEO for every single decision." Even for the simple stuff like "Check the address," you'd hire the CEO. It works, but it costs a fortune and takes a long time.

The paper AgentFloor asks a simple question: Do we really need the CEO for every single task? Or can we hire a smart, fast, and cheap intern for the routine jobs, saving the CEO only for the really hard problems?

To find out, the researchers built a giant, 30-step obstacle course (a benchmark) designed to test how well different AI "workers" can use tools. They tested 16 different AI models of varying sizes (from tiny "interns" to massive "CEOs") against the current top-tier AI (GPT-5).

Here is what they found, explained through a few simple analogies:

1. The "Ladder" of Difficulty

The researchers organized the tasks into a 6-rung ladder. As you climb up, the jobs get harder:

Rungs 1 & 2 (The Ground Floor): Simple instructions and using one tool (like looking up a phone number).
Rungs 3 & 4 (The Middle): Chaining two tools together or making a decision based on a result (like "If the price is high, call the manager").
Rungs 5 & 6 (The Summit): Complex planning where you have to remember rules for a long time and coordinate many steps without getting lost.

2. The Results: Who Can Climb What?

The study revealed a clear "floor" where small models stop being able to do the job.

The Ground Floor (Rungs 1 & 2): The small, cheap models (the "interns") are actually better or just as good as the expensive CEO. They can follow simple instructions and use single tools perfectly. In fact, the small models were so fast and cheap that they could do the same amount of work for 15 times less money and 2.5 times faster.
The Middle Rungs (Rungs 3 & 4): The small models are still doing a great job. They are very close to the CEO's performance. The study couldn't say they were exactly equal with 100% statistical certainty, but they were close enough to be useful.
The Summit (Rungs 5 & 6): This is where the gap appears. When the task requires long-term planning and remembering many rules at once, the small models start to stumble. They get confused, run out of "steps" (like a runner getting tired), or start making up tools that don't exist. The big CEO (GPT-5) is still better here, but even the CEO isn't perfect; it still fails quite a bit on these hardest tasks.

3. The "Magic Fix" Didn't Work

The researchers tried to give the small models a "cheat sheet" (a special prompt) to help them climb the higher rungs, hoping it would make them as smart as the CEO.

The Result: It didn't work universally. A trick that helped one specific model sometimes made another model worse. It's like giving a specific pair of running shoes to a marathon runner; it might help that runner, but it could trip up someone else. There is no single "magic button" that makes a small model instantly as smart as a big one for complex planning.

4. The Big Takeaway: The "Smart Routing" Strategy

The paper suggests a new way to build AI systems, which they call routing.

Instead of using the expensive, slow, super-smart AI for everything, you should build a system that acts like a smart manager:

Send the easy, routine jobs (checking data, simple lookups) to the small, cheap models. They are fast, cheap, and just as good at this work.
Reserve the big, expensive AI only for the rare, complex jobs that require deep planning and long-term memory.

The Bottom Line

You don't need a Ferrari to drive to the grocery store; a reliable bicycle is faster and cheaper for that trip.

The paper proves that for the vast majority of "routine" AI tasks (like checking a database or using a single tool), small, open-source models are already good enough. You only need the massive, expensive "frontier" models for the very top tier of complex planning, and even then, they aren't perfect yet.

By using the right "worker" for the right job, companies can save a massive amount of money and speed up their systems without losing quality.

1. Problem Statement

Production agentic systems frequently make numerous model calls per user request. A significant portion of these calls are short, structured, and routine (e.g., lookups, record extraction, single submissions). Currently, the industry default is to route all such calls to expensive, large "frontier" models (e.g., GPT-5).

The Core Question: Existing evaluations do not directly answer which parts of an agent workflow truly require frontier intelligence and which can be handled by smaller, open-weight models.
The Gap: Single-turn benchmarks (e.g., BFCL) ignore sequential dependencies, while complex multi-step suites (e.g., SWE-Bench) are confounded by API drift, web grounding, and data contamination. There is no controlled map of where small models match frontier performance and where they diverge.

2. Methodology: The AgentFloor Benchmark

The authors introduce AgentFloor, a deterministic, 30-task benchmark designed to isolate cognitive demands in tool use.

Structure: The benchmark is organized as a six-tier capability ladder (A0 to E), where each tier introduces exactly one new cognitive demand:
- A0: Instruction following (no tools).
- A: Single-tool invocation.
- B: Sequential 2-tool chaining (output of Tool 1 feeds Tool 2).
- C: Conditional branching based on intermediate results.
- D: Multi-source synthesis with conflict recovery.
- E: Long-horizon planning under persistent constraints (8–12 steps).
Environment: Tasks run in a fixed, abstract tool environment with an in-memory fixture database. There are no filesystems, live APIs, or time-varying states, eliminating environmental noise and pretraining contamination.
Models Evaluated: 16 open-weight models ranging from 0.27B to 32B parameters (including Gemma, Qwen, Mistral, Granite, Ministral) alongside GPT-5.
Evaluation Protocol:
- Runs: 16,542 scored runs.
- Scoring: Binary Pass/Fail based on four checker families (final answer match, submission validation, trajectory satisfaction, and no forbidden behaviors like hallucinated tools).
- Statistical Rigor: Uses paired bootstrap analyses (10,000 resamples) and Two-One-Sided Test (TOST) for equivalence at a pre-registered ±10 percentage point (pp) margin.
- Failure Taxonomy: Failures are classified into a strict cascade (F1: Hallucinated tool, F4: Step-budget exhausted, F5: Early resignation, etc.).

3. Key Contributions

AgentFloor Benchmark: A controlled, deterministic 30-task ladder that isolates specific cognitive demands in agentic tool use, removing confounds like API drift.
Capability-and-Cost Map: A pre-registered comparison showing that the best open-weight model (Gemma 4:26b) achieves aggregate parity with GPT-5 while being significantly cheaper and faster.
Failure-Mode Analysis: A granular analysis revealing that the "long-horizon gap" is not universal; different models fail for different reasons (e.g., running out of steps vs. resigning early), meaning interventions are model-specific rather than universal.

4. Key Results

A. Aggregate Parity and Cost Efficiency

Performance: Across all 30 tasks, Gemma 4:26b is statistically equivalent to GPT-5 (paired $\Delta = +0.4$ pp, 90% CI $[-4.0, +5.1]$ ).
Cost: At matched aggregate accuracy (~60% TCR), Gemma 4:26b is 15× cheaper than GPT-5 on self-hosted Macs and 3× cheaper on cloud GPUs.
Latency: Gemma 4:26b is 2.5× faster per passed task than GPT-5.
Pareto Frontier: Open-weight models occupy the entire Pareto frontier of cost-per-passed-task at every reliability point measured.

B. Tier-by-Tier Breakdown

A0 (Instruction): Open-weight models (e.g., Gemma 4:26b) strictly outperform GPT-5 (100% vs 80%). GPT-5 failures were often due to over-paraphrasing technical vocabulary.
A (Single Tool): Open-weight models are formally equivalent to GPT-5.
B, C, D (Chaining, Branching, Synthesis): Point estimates are close to GPT-5 (within 9 pp), but sample sizes were insufficient to formally certify equivalence at the ±10 pp margin.
E (Long-Horizon Planning): This is the only tier where GPT-5 is strictly superior (10% pass rate vs. 0% for Gemma 4:26b). However, neither model reaches "practitioner-relevant" reliability (>80%) on this tier.

C. The "Cliff" and Model Sizing

Small Models Suffice for Routine Work: For tiers A0 and A, models as small as 0.6B–4B parameters are sufficient to clear 80–90% reliability targets.
The B→C Transition: There is a sharp drop in performance moving from sequential chaining (B) to branching (C). No zero-shot open-weight model in the corpus cleared the 80% reliability bar on tiers C, D, or E.
Non-Monotonic Scaling: Performance does not strictly scale with parameter count. For example, a 4B model (Nemotron) outperformed a 26B model on Tier D, and a 14B model was dominated by its 8B sibling in the same family.

D. Intervention Analysis (Why the Gap Exists)

The authors tested interventions to close the Tier E gap but found no universal fix:

Explicit-Submission Prompt: Lifted Ministral-3:8b on a specific task from 0% to 100% but had zero effect on GPT-5 or Gemma 4:26b.
Step-Budget Doubling: Helped GPT-5 on one specific task (E1) but was null on others.
Reasoning Mode: Disabling reasoning helped the 32B Qwen model on Tier B but hurt the 8B and 14B versions.
Structured Prompts (Plan/Execute/Submit): This "obvious" universal lever degraded performance across all tested models, causing them to plan but fail to execute.
Failure Signatures:
- GPT-5 fails primarily via F5 (Early Resignation): It plans but gives up.
- Gemma 4:26b fails via F4 (Step Exhaustion) and F1 (Hallucinated Tools): It runs out of steps or invents tools.
- Conclusion: Because the failure mechanisms differ, a single intervention cannot fix both.

5. Significance and Design Principles

The paper provides a concrete, data-driven guide for agentic system routing:

Route Routine to Small Models: The vast majority of agent calls (A0, A, and likely B) can be handled by small, open-weight models (<5B parameters) with no measurable loss in accuracy and massive gains in cost and latency.
Reserve Frontier for Long-Horizon: Frontier models should be reserved strictly for the narrow class of tasks requiring sustained coordination and constraint tracking (Tier E), though even there, reliability remains low.
Interventions are Model-Specific: There is no "silver bullet" prompt or architecture tweak that universally fixes long-horizon planning. Routing strategies must be tailored to the specific failure modes of the chosen model.

Final Takeaway: The "tool use ladder" has a clear boundary. Small open-weight models are already sufficient for the broad base of structured, short-horizon agentic work, challenging the default deployment strategy of routing everything to frontier models.