💬 NLP

Enrich-Retrieve-Rank: Scaling Capability Discovery Beyond In-Context Routing

This paper introduces "Enrich-Retrieve-Rank," a scalable capability discovery pipeline that replaces inefficient in-context routing with an offline enrichment and online retrieve-then-rank approach, demonstrating significantly higher accuracy and lower costs than baseline methods as agent ecosystems grow from hundreds to thousands of components.

Original authors: Nazib Sorathiya, Daniel Zhang, Bardiya Akhbari

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nazib Sorathiya, Daniel Zhang, Bardiya Akhbari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a new generation of systems has emerged that does not just answer questions but performs tasks. These systems, often called agents, act as digital assistants that can browse the web, write code, or manage complex workflows. To do this, they rely on a vast library of digital tools, skills, and other agents, much like a human worker relies on a toolbox filled with hammers, screwdrivers, and specialized software. As these ecosystems grow, they now contain thousands of distinct capabilities. The central challenge for these intelligent systems is discovery: when faced with a specific task, how does the system know which tool to pick from a library that might contain thousands of options? For a long time, the standard approach was to simply show the entire list of available tools to the artificial intelligence and ask it to choose. This method worked well when the list was short, but as the libraries expanded into the thousands, the systems began to struggle, often failing to find the right tool or wasting time and resources trying the wrong ones.

A team of researchers at Amazon AGI set out to solve this scaling problem by reimagining how these systems search for capabilities. They treated the problem not as a simple choice, but as a search engine query. Instead of forcing the artificial intelligence to read through a massive, unorganized list of every available tool, they built a pipeline that first organizes the information and then retrieves the most relevant options before making a final decision. The process begins with an offline step where the system takes the sparse, often brief descriptions of each tool and rewrites them into rich, detailed profiles. This is done once when a tool is added to the library, creating a searchable record that includes a summary, specific keywords, and examples of how the tool is used. When a user asks a question, the system does not ask the artificial intelligence to guess from the whole library. Instead, it quickly searches this enriched index to pull up a shortlist of the most likely candidates, and then uses a final, focused check to rank them.

The researchers tested this approach against the traditional method of showing the entire list to the artificial intelligence, known as in-context routing. They found that as the library of tools grew, the traditional method collapsed. When the library contained only ten tools, the system was correct about eighty-five percent of the time. However, as the number of tools grew to nearly seven thousand, the accuracy of the traditional method plummeted to just twelve percent. The system became overwhelmed by the sheer volume of information it had to process at once. In contrast, the new search-and-rank pipeline handled the growth much more gracefully. While its accuracy also decreased as the library grew, it remained significantly higher, dropping to thirty-nine percent at the largest scale. Crucially, the researchers discovered a tipping point around five hundred tools. Below this number, the old method of showing the whole list still worked well. Above it, the new search-based approach became essential, outperforming the old method by a wide margin.

The study also revealed that the new system was far more efficient. By retrieving a shortlist first, the system used roughly half the computing resources required by the traditional method when compared to a system that searched for candidates one by one. When compared to the method that showed the entire list, the new approach reduced costs by a factor of seventy. This efficiency is vital because every time an artificial intelligence tries a tool and fails, it consumes computing power and time, and in some cases, it might even trigger an action on an untrusted system just to learn what it does. The new pipeline avoids these wasted attempts by filtering out irrelevant options before any action is taken.

A key finding of the research was that the improvement came primarily from the retrieval step, not the final ranking step. The system's ability to find the correct tool in the initial search was the main bottleneck. Even when the search engine found the right tool, the final ranking step was highly reliable, correctly placing the best option at the top of the list between seventy and eighty-seven percent of the time, regardless of how large the library was. This suggests that the future of these systems depends less on making the final decision-maker smarter and more on building better search engines that can handle massive libraries of tools. The researchers also noted that this approach works best when the initial descriptions of the tools are sparse or incomplete. For libraries that already have very detailed descriptions, the extra step of rewriting the information provided little benefit. However, for the vast, growing libraries of real-world tools where descriptions are often brief, this enrichment step was critical for success.

The team deployed this pipeline as the default discovery layer for a large-scale multi-agent platform in production. In this real-world environment, the system manages thousands of agents, tools, and skills simultaneously. The results confirmed the laboratory findings: the search-based approach allowed the system to scale effectively where the old methods failed. The researchers emphasized that this is not a theoretical exercise but a practical solution to a problem that arises as artificial intelligence systems become more complex and capable. By treating capability discovery as a search problem rather than a selection problem, they have provided a blueprint for how these systems can continue to grow without losing their ability to find the right tools for the job. The work demonstrates that as artificial intelligence ecosystems expand, the architecture of how they find information must evolve, moving away from simple lists and toward sophisticated, layered search processes that can handle the complexity of the modern digital world.

Technical Summary: Enrich-Retrieve-Rank: Scaling Capability Discovery Beyond In-Context Routing

Problem Statement

Modern agent ecosystems increasingly rely on registries containing thousands of components (Models, Agents, Tools, and Skills, collectively MATS). Current discovery mechanisms rely on in-context routing, where a Large Language Model (LLM) reads a registry representation (names, hints, or descriptions) within its context window, selects a candidate, invokes it, and retries upon failure.

This approach suffers from severe scalability issues:

Context Limits: As registries grow (e.g., to 7,278 tools), they exceed the context window of standard LLMs, forcing truncation (e.g., alphabetical) which silently drops valid candidates.
Cost and Latency: Every incorrect invocation incurs token costs and latency. In a "trial-and-error" or "search-and-pick" loop, the cost scales linearly or worse with registry size.
Accuracy Collapse: The paper demonstrates that as registry size ( $N$ ) increases from 10 to 7,278, the top-1 accuracy (Match@1) of in-context routing collapses from 0.85 to 0.12.

The authors frame capability discovery not as a selection problem within a prompt, but as an information retrieval (IR) problem over a capability registry, analogous to web search.

Methodology: The Enrich-Retrieve-Rank Pipeline

The proposed solution decouples discovery into an offline enrichment phase and an online retrieve-then-rank pipeline.

1. Offline Enrichment (Registration Time)

Instead of relying on sparse metadata (e.g., a short name or one-line hint) at query time, the system performs a one-time transformation for each capability upon registration.

Process: An LLM rewrites sparse metadata into a rich, structured profile containing five typed fields:
- Capability summary
- Action-verb-led description
- Differentiating keywords
- Positive usage examples
- Negative usage examples
Production Additions: For internal registries, the profile also includes a trust score and self-reported capability tags.
Indexing: These enriched profiles are indexed both lexically (for BM25) and densely (using embeddings like BGE-large-en-v1.5 or Amazon Titan Embed V2).

2. Online Retrieve-Then-Rank (Query Time)

The pipeline operates in two stages to return a ranked shortlist without invoking any candidates:

Phase 2: Retrieve: Given a user query, the system retrieves the top- $k$ candidates (where $k \in \{15, 25\}$ ) from the enriched index using lexical (BM25), neural, or hybrid retrieval.
Phase 3: Rank: A reranker (LLM) scores the retrieved candidates. The final score is a weighted combination of:
- LLM Score (50%): The LLM's assessment of the candidate's relevance based on the enriched profile.
- BM25 Score (5%): Lexical overlap.
- Quality Score (30%): Normalized offline trust score (production only).
- Intent Score (15%): Match between inferred query type and candidate's self-reported tags (production only).
Output: A ranked shortlist is returned to the orchestrator, which then selects and invokes the final candidate.

Key Contributions

The paper positions itself as a systems and scaling study rather than a novel model proposal. Its primary contributions are:

Scaling Curve Analysis: The authors map the full degradation curve of capability discovery from $N=10$ to $N=7,278$ . They identify a crossover point around $N=500$ (in the Nova Micro sweep) where in-context routing fails and the retrieve-then-rank pipeline becomes superior.
Error Decomposition: The study decomposes performance into retrieval recall and reranker conditional accuracy.
- The reranker maintains a stable conditional accuracy of 0.70–0.87 across all registry sizes.
- The majority of failures (approx. 70% in large registries) originate at the retrieval stage, not the ranking stage.
Production-Efficiency Comparison: The authors compare their pipeline against baselines grounded in deployed routing strategies (Full-Ctx, Search&Pick, Trial&Err), measuring accuracy, token cost, latency, and dollar cost.

Experimental Results

The pipeline was evaluated across three capability types (Tools, Agents, Skills) and five dataset combinations, including the large-scale ToolRet benchmark ( $N=7,278$ tools).

Accuracy at Scale: On the Tools-ToolRet benchmark ( $N=7,278$ ), the pipeline (Ours+Titan) achieves a Match@1 of 0.397. This leads the "Search&Pick" baseline (0.332) by 6.5 percentage points (pp). In contrast, Full-Ctx routing is infeasible at full scale due to context limits; on a 500-item subsample, it achieves a Match@1 of 0.166, and performance collapses further as the registry grows beyond the context window.
Cost Efficiency:
- The pipeline is approximately 2× cheaper than Search&Pick.
- It reduces costs by 70× compared to Full-Ctx routing ($0.066 vs. $4.48 per 1,000 queries).
- Latency is bounded, requiring only one LLM call for reranking.
Enrichment Impact:
- On well-documented public benchmarks (e.g., AppWorld, MCP), enrichment is neutral-to-negative, as the metadata is already rich.
- However, in a controlled stress test where metadata was degraded to "name-only," enrichment improved Match@1 by +25.6 pp and Recall@15 from 0.134 to 0.467. This confirms enrichment's value scales with metadata sparsity.
Generalization: A single configuration (same enrichment template, retriever, and weights) was used across all capability types without per-type tuning, demonstrating cross-type robustness.

Significance and Claims

The paper claims that capability discovery must shift from in-context routing to an IR-based architecture to support the growth of agent ecosystems.

Architectural Necessity: The authors state that for registries exceeding ~500 entries, retrieve-then-rank is an "architectural requirement, not a preference," as single-stage LLM routing degrades rapidly.
Production Deployment: The pipeline is currently deployed as the default capability-discovery layer for a large-scale multi-agent platform at Amazon, handling over 500 entries across all MATS types.
Modest Claims on Enrichment: The authors explicitly note that enrichment is a production design choice to handle sparse metadata, not a source of benchmark gains on well-documented public data.
Limitations: The study acknowledges that current benchmarks (like AppWorld and MCP) often have small registry sizes or query sets that paraphrase descriptions, limiting the ability to independently validate cross-type gains at scale. Additionally, the "Quality" and "Intent" signals rely on metadata (trust scores, type tags) absent from public benchmarks.

In summary, the paper demonstrates that by treating capability discovery as a search problem with offline enrichment and a two-stage retrieve-rank pipeline, systems can scale to thousands of components with significantly higher accuracy and lower cost than traditional in-context routing.