💬 NLP

Scaling Enterprise Agent Routing: Degradation, Diagnosis, and Recovery

This paper investigates how routing accuracy in enterprise LLM assistants degrades as tool catalogs scale from 10 to 110 agents, identifying retrieval and confusion gaps as primary causes of failure and demonstrating that embedding-based shortlisting effectively recovers significant F1 performance across multiple frontier models.

Original authors: Kellen Gillespie, Robyn Perry

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kellen Gillespie, Robyn Perry

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, busy office. You have a team of 110 specialized employees (agents) and 584 specific skills (tools) they can use. Your job is to take a request from a client (like "I need to send an email") and instantly pick the one right employee to handle it.

This paper is a report on what happens when that office gets too big, and how the manager (an AI) starts to make mistakes.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Too Many Choices" Trap

When the office was small (say, 10 employees), the manager was great at picking the right person. But as they added more people, the manager started to get confused.

The Analogy: Imagine trying to find a specific needle in a haystack. When the haystack is small, it's easy. When it grows to the size of a mountain, you start missing needles you should have found.
The Result: As the catalog of tools grew from 10 to 110, the AI's ability to find the correct tool dropped significantly (by about 16–23%). It wasn't that the AI started picking wrong tools more often; it was that it started missing the right tools entirely.

2. Why Did It Fail? Two Types of Mistakes

The researchers broke down the failure into two distinct problems, like two different reasons a detective might fail to solve a case:

The "Retrieval Gap" (The Librarian Problem): The AI couldn't even see the right tool in the list. It was like a librarian who couldn't find the book on the shelf because the library was too big and messy. This is the part the researchers could fix.
The "Confusion Gap" (The Twin Problem): Even if the AI did see the right tool, it still got confused. Why? Because the office has many "twins."
- Example: You have Gmail, Outlook, and Yahoo Mail. You have "Improve," "Paraphrase," and "Proofread" tools. They all do almost the same thing. Even with a perfect list, the AI struggled to decide which "twin" was the best fit. This confusion caused a permanent drop in performance that couldn't be fixed just by listing tools better.

3. The Solution: The "Shortlist" Filter

The researchers tested a simple fix: Don't show the AI the whole list of 584 tools. Instead, use a quick filter (like a smart search bar) to find the top 20 most likely candidates first, and then ask the AI to pick the winner from that small group.

The Analogy: Instead of asking a judge to pick a winner from 584 contestants, you first have a scout pick the top 20. Then, the judge picks the winner from just those 20.
The Result: This "Shortlisting" method worked like magic. It recovered about 10–17% of the lost performance. It didn't fix the "Twin Problem" (confusion), but it completely solved the "Librarian Problem" (missing the right tool).

4. What Didn't Work (and What Did)

The team tried different ways to build this "scout" (the filter):

The Winner: Using Embeddings (a type of AI that understands the meaning of words, not just the spelling). This was the best method. It understood that "send a message" and "email the team" are similar, even if the words are different.
The Loser: Keyword Search (looking for exact word matches). This failed miserably because in a big office, everyone uses the same words (like "email" or "schedule"), so searching for those words didn't help distinguish between the different tools.
The "Pack" Approach: Grouping tools into categories (like "Email Tools" vs. "Writing Tools") and picking a category first didn't work as well as picking individual tools directly.

5. Real-World Testing

The researchers didn't just use computer-generated test questions. They also tested this on 1,435 real requests from actual users.

Real users are messy: they misspell words, use slang, and don't say exactly what they mean.
The results held up: Even with messy real-world data, the "Shortlist" method still boosted performance significantly, proving this isn't just a lab trick—it works in the real world.

The Bottom Line

When you scale up an AI system with too many tools, it gets overwhelmed and starts missing the right answers.

The Fix: Don't let the AI look at everything at once. Use a smart filter to narrow it down to a shortlist of 20 options first.
The Limit: Even with a shortlist, the AI will still get confused by tools that do very similar things (like different email apps). That part is harder to fix, but the "Shortlist" method solves the biggest part of the problem.

Technical Summary: Scaling Enterprise Agent Routing

Problem Statement

As Large Language Model (LLM) assistants evolve into orchestration layers, they increasingly route user requests to growing libraries of specialized tools and agents. While platforms like OpenAI and Anthropic have introduced mechanisms (e.g., namespace-based search, BM25 retrieval) to manage this growth, the specific mechanisms of performance degradation as catalogs scale remain under-characterized.

This paper addresses the critical question: How does single-step routing accuracy degrade as the agent catalog scales from a manageable size to an enterprise-scale library? The authors focus on a production environment characterized by "dense semantic overlap," where functionally similar tools (e.g., multiple email clients, writing assistants, or project trackers) create significant routing ambiguity, a scenario less common in API-centric benchmarks.

Methodology

Experimental Setup

The study utilizes a real-world, deployed enterprise productivity catalog comprising 110 agents and 584 individual tools. The evaluation spans scale points from 10 to 110 agents (51 to 584 tools).

Models: Three frontier models were evaluated: GPT-5.1, GPT-5.4 (OpenAI), and Claude Sonnet 4.5 (Anthropic).
Data:
- Synthetic Queries: 4,105 queries generated by GPT-4o, categorized as explicit (naming the tool) and implicit (describing the need). The study focuses primarily on implicit queries, which represent realistic production traffic.
- Production Queries: 1,435 human-annotated utterances sampled from live traffic, annotated by three linguists to establish a dynamic ground truth that accounts for semantic overlap.
Metrics: Multi-label precision, recall, and F1 score, computed against a dynamic valid set (target tool + any also-valid tools present in the specific scale fold).

Diagnostic Framework

The authors employ an Oracle Analysis to decompose routing failures into two distinct gaps:

Retrieval Gap: The performance loss caused by the model's inability to surface the correct tool from the full catalog.
Confusion Gap: The performance loss that persists even when the correct tool is guaranteed to be present (the "oracle ceiling" drops as the catalog grows).

Interventions

The study evaluates embedding-based shortlisting as a primary intervention. This involves using a dense retriever (text-embedding-3-large) to pre-filter the full catalog down to a shortlist of $k=20$ tools before the LLM router makes its final decision. This is compared against:

Flat routing (no shortlisting).
Platform-native tool search (OpenAI's namespace search).
Pack-level approaches (hierarchical routing, pack-level embedding).
Fine-tuned retrievers (ToolRet-e5).
Lexical retrieval (BM25).

Key Results

1. Scaling Degradation

Routing accuracy degrades significantly as the catalog expands.

F1 Drop: On implicit queries, F1 scores drop by 16–23 percentage points across all models as the catalog grows from 51 to 584 tools.
Recall-Driven: The degradation is primarily driven by a collapse in recall (dropping from ~55% to ~37%), while precision drops more moderately. The model increasingly misses valid tools rather than selecting invalid ones.
Oracle Ceiling: Even with perfect retrieval (an oracle shortlist), the F1 ceiling drops by 10 percentage points (from 79% to 69%). This reveals a confusion gap where the model struggles to distinguish between semantically similar tools even when they are presented together.

2. Effectiveness of Shortlisting

Embedding-based shortlisting ( $k=20$ ) successfully recovers a significant portion of the lost performance.

Recovery: Shortlisting recovers +10–11 percentage points in F1 at full scale across all three models and two providers.
Production Validation: On real-world human-annotated traffic, the recovery is even more pronounced, ranging from +10 to +17 percentage points, despite the absolute baseline performance being 10–15 points lower than synthetic data.
Error Reduction: Shortlisting reduces routing misses from 31% to 10% at full scale. The trade-off is a stable 9% shortlister miss rate (cases where the correct tool is not included in the top 20).

3. Comparative Performance

Tool-Level vs. Pack-Level: Tool-level retrieval consistently outperforms pack-level approaches (hierarchical routing, pack-level embedding) by 2–4 percentage points. Pack-level methods often dilute the candidate set with irrelevant sibling tools.
Dense vs. Lexical: Dense embedding retrieval (52.5% F1) significantly outperforms lexical matching (BM25, 32.8% F1) and fine-tuned retrievers trained on API-centric data (48.7% F1). BM25 fails because enterprise tools share high-vocabulary overlap.
Model Consistency: While stronger models (e.g., GPT-5.4, Sonnet) provide a higher baseline, they follow the same degradation curve. Shortlisting provides a consistent delta improvement regardless of the base model strength.

Significance and Claims

The paper claims to provide a controlled, production-grounded diagnosis of the "scaling law" for agent routing. Its primary contributions are:

Decomposition of Failure: It distinguishes between retrieval errors (the model can't find the tool) and confusion errors (the model can't choose between similar tools). This clarifies that while better retrieval can solve the majority of scaling issues, a fundamental confusion gap remains that retrieval alone cannot close.
Validation of Shortlisting: It demonstrates that embedding-based shortlisting is a highly effective, scalable intervention that recovers ~10–17pp of F1, making it a viable strategy for enterprise systems with hundreds of tools.
Limitations of Current Platforms: The study suggests that platform-native search features (like OpenAI's namespace search) provide only partial relief and plateau at larger catalog sizes, whereas custom embedding shortlisting continues to scale effectively.
Domain Specificity: The authors note that these findings are specific to domains with dense semantic overlap (like enterprise productivity). Domains with functionally distinct tools may degrade more slowly, but the qualitative pattern of recall-driven degradation is likely transferable.

The paper concludes that while the retrieval gap is addressable through better candidate selection (shortlisting), the confusion gap (at least 10pp) requires architectural or prompt-level innovations beyond simple retrieval, such as description deduplication or multi-turn clarification.