🤖 AI

Indexing the Unreadable: LLM-Native Recursive Construction and Search of Service Taxonomies

To address the context window limitations and "Lost-in-the-Middle" phenomenon in the emerging Internet of Agents, this paper proposes A2X, an LLM-native system that dynamically organizes services into a hierarchical taxonomy and navigates it layer-by-layer to significantly improve retrieval accuracy while drastically reducing token consumption compared to existing baselines.

Original authors: Wei Zheng, Yang Yan, Yiyang Shao, Jinyang Li, Zeze Chang, Yukuang Jia, Qiming Mao, Chihyung Wang, Jingbin Zhou

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wei Zheng, Yang Yan, Yiyang Shao, Jinyang Li, Zeze Chang, Yukuang Jia, Qiming Mao, Chihyung Wang, Jingbin Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Too Many Options" Dilemma

Imagine you are a very smart assistant (an AI agent) trying to help a user. To do your job, you have access to a massive library of tools—thousands of different "apps" or "services" like booking flights, checking bank balances, or translating languages.

The problem is that your brain (the AI's "context window") can only hold a limited amount of information at once.

The Old Way: If you tried to read the descriptions of all 1,000 tools at once to find the right one, you would get overwhelmed. It's like trying to read a whole encyclopedia in one sitting. Even if you could read it all, your brain tends to ignore the middle pages and focus only on the beginning and the end. This is called the "Lost-in-the-Middle" effect. You might miss the perfect tool because it was buried in the middle of the list.
The Alternative: Some people try to use a "magic index" (embedding-based search) that guesses which tools are similar to your question. But this is like using a blurry map; it often misses the specific, unique tools you actually need because it doesn't truly understand the meaning of the words, just their shape.

The Solution: A2X (The Smart Librarian)

The authors created a system called A2X (Agent-to-Anything). Instead of dumping the whole library on the AI's desk, A2X acts like a super-organized librarian who builds a custom filing system and guides the AI step-by-step.

Here is how it works, using a simple analogy:

1. Building the Map (The Taxonomy)

Imagine you have a pile of 2,000 random tools. A human would take days to sort them. A2X uses the AI itself to do the sorting, but it does it in small, manageable chunks.

Recursive Splitting: Instead of asking the AI to sort 2,000 items at once (which causes the "overwhelm"), A2X asks it to sort 50 items, then another 50, and so on.
The Result: It builds a hierarchical tree (like a family tree or a website menu).
- Level 1: Big categories (e.g., "Travel," "Finance," "Health").
- Level 2: Sub-categories (e.g., under "Travel": "Flights," "Hotels").
- Level 3: Specific tools.
No Human Needed: The AI builds this map automatically from scratch, without a human needing to write a manual.

2. Finding the Tool (Progressive Disclosure)

When a user asks, "I need to book a flight to Tokyo," the AI doesn't look at the whole library. It follows the map:

Step 1: The AI looks at the top level. It sees "Travel," "Finance," and "Health." It instantly ignores "Finance" and "Health" and focuses only on "Travel."
Step 2: It zooms into "Travel." It sees "Flights," "Hotels," and "Car Rentals." It ignores the others and focuses only on "Flights."
Step 3: It looks at the specific flight tools. Now it only has to choose between a handful of options, not 2,000.

This is called Progressive Disclosure. The AI only ever sees a small, relevant list of options at any given time. It never gets overwhelmed, and it never misses the tool in the middle of a long list.

Why It's Better (The Results)

The paper tested this system against the old ways and found:

Better Accuracy: A2X found the right tools 20% more often than the best "magic index" systems (embedding search). It also found them 6% more often than just dumping the whole list into the AI's brain.
Cheaper and Faster: Because the AI only reads small lists instead of the whole library, it uses 9 times less computer memory (tokens) to get the same (or better) result.
Works in Other Languages: The system worked just as well in Chinese as it did in English, because the AI understands the meaning of the words, not just the translation.

The Bottom Line

The paper argues that as we enter an era where AI agents need to use thousands of tools, we can't just feed them all the data at once. Instead, we need to teach them how to navigate a structured map. A2X is a system that automatically builds that map and guides the AI through it, one small step at a time, ensuring it finds the right tool without getting lost or running out of brainpower.

Technical Summary: Indexing the Unreadable (A2X)

1. Problem Statement

The paper addresses a critical bottleneck in the emerging "Internet of Agents" (IoA): the structural mismatch between the rapidly expanding population of LLM-callable services (MCP servers, A2A endpoints, skills) and the limited effective context window of Large Language Models (LLMs).

Two primary failure modes arise when agents attempt to discover services from large registries:

Token Bloat: Concatenating thousands of service descriptions into a single prompt exceeds context windows or incurs prohibitive token costs. For example, a registry of 1,000 tools can occupy ~500k tokens.
Lost-in-the-Middle (LIM): Even when the context window is nominally large enough, LLMs systematically under-attend to information in the middle of long inputs. Recent benchmarks confirm that retrieval accuracy does not scale linearly with context length; effective context is the binding constraint, not raw window size.

Existing solutions face trade-offs:

Full-context dumping: Fails due to token limits and LIM degradation.
Embedding-based retrieval: While token-efficient, dense vector search struggles with cross-vocabulary, multi-constraint, and long-tail queries, often missing relevant services due to a lack of deep semantic understanding.
Manual taxonomies: High maintenance costs and structural lag make them unsuitable for dynamic, rapidly growing service populations.

The core research question is: How should a large, dynamic registry be organized and selectively disclosed to an LLM so that every decision is made over a short, highly relevant candidate set, while the registry as a whole remains discoverable?

2. Methodology: A2X (Agent-to-Anything)

The authors propose A2X, an LLM-native service discovery system that reframes the problem as context management via progressive disclosure. Instead of presenting the entire registry, A2X organizes services into a hierarchical taxonomy and walks it layer-by-layer at query time.

A. Automatic Taxonomy Construction (Offline)

A2X constructs the taxonomy autonomously from raw service names and descriptions using a Breadth-First Search (BFS) algorithm (Algorithm 1), avoiding the need for human-curated ontologies.

Context-Aware Splitting: To prevent context overflow during construction, nodes exceeding a threshold ( $\theta_{kw} = 500$ services) are processed via a keyword-first strategy. A KeywordExtractor batches services to generate a frequency-weighted keyword list, which is then fed to an LLM designer. This ensures the designer never sees raw descriptions at scale.
Single-Axis Constraint: To prevent "axis mixing" (e.g., mixing functional domains with technical approaches), the design prompt enforces that sibling categories within a node must share a single classification axis. A preference chain prioritizes user-facing functional domains (e.g., "Travel") over operational or technical axes.
Boundary Refinement: To handle semantic ambiguity between sibling categories (e.g., "Finance" vs. "Business"), the system employs a classification-refinement loop. If services are misclassified or unassigned, the system feeds these edge cases back to the designer to refine category definitions and "NOT here" boundary clauses.
Cross-Domain Handling: To address services that legitimately span multiple top-level domains (which a strict tree would miss), a CrossDomainAssigner performs a multi-parent pass, allowing services to appear under multiple relevant leaves.

B. Progressive-Disclosure Search (Online)

At query time, A2X navigates the taxonomy recursively (Algorithm 2), ensuring every LLM call processes only a small candidate set.

Parallel Recursive Descent (CategoryNavigator): Starting from the root, the LLM is presented with the children of the current node (typically ~8 options) and asked to select relevant branches. The unselected subtrees are pruned immediately.
Service Selection (ServiceSelector): Once leaf nodes are reached, the system collects candidate services. It deduplicates services and merges small groups (using a least-common-ancestor heuristic) to ensure group sizes remain within the LLM's effective context budget.
Modes: The system supports different search modes (get_all, get_important, get_one) by adjusting the system prompt to trade off between recall and precision, but the underlying navigation mechanism remains the same.

The total token cost per query grows sub-linearly with the registry size $N$ , as the number of calls is proportional to the tree depth ( $d$ ) and branching factor ( $b$ ), rather than $N$ itself.

3. Key Contributions

Reframing Service Discovery: The authors reframe agent-side service discovery as a context management problem, proposing LLM-native progressive disclosure as the solution to decouple effective context scarcity from registry size.
A2X System: A fully autonomous pipeline that:
- Automatically constructs a hierarchical taxonomy from raw service descriptions using recursive splitting and refinement loops.
- Navigates this taxonomy layer-by-layer at query time, ensuring every LLM call sees only a small, relevant candidate set.
Empirical Dominance: On the ToolRet benchmark (1,839 services, 1,714 queries), A2X achieves:
- 92.6% Hit Rate, outperforming full-context dumping by 6.2 absolute points and the state-of-the-art embedding-based baseline by >20 points.
- 1/9th the prompt-token cost compared to full-context dumping.
- Strong cross-lingual robustness, maintaining similar performance on a Chinese variant (ToolRet_CN) without language-specific engineering.

4. Experimental Results

Search Efficiency (Study 1): Compared to a pure-LLM baseline that dumps all 1,839 services into a single prompt (67,619 tokens/query), A2X achieves higher Hit Rate (92.6% vs. 86.4%) and Recall (89.2% vs. 84.7%) while consuming only 7,069 tokens/query.
Construction Quality (Study 2): Ablation studies show that the recursive BFS construction with keyword compression, refinement loops, and single-axis constraints is essential. A "one-shot" LLM build (reading all descriptions at once) achieves only ~60% Hit Rate, whereas the recursive approach reaches 92.6%.
Embedding Comparison (Study 3): A2X significantly outperforms dense vector retrieval (MiniLM, BGE-M3, etc.) and query-rewriting baselines (MCP-Zero). Embedding methods suffer from insufficient recall (e.g., 61.8% HR for MiniLM vs. 92.6% for A2X), particularly in complex, multi-constraint scenarios.
Model Robustness: Experiments with newer models (DeepSeek V4) confirm that while long-context capabilities improve full-context baselines, A2X maintains a cost advantage and, in many cases, an accuracy advantage, proving the method is not tied to a specific model generation.

5. Significance and Claims

The paper claims that LLM-native discovery is the natural successor to embedding-based retrieval as inference costs fall.

Paradigm Shift: Unlike embedding-based methods that compress semantic meaning into fixed-dimensional vectors, A2X leverages the LLM's native ability to perform semantic reasoning and disambiguate complex queries.
Scalability: By interposing a hierarchical taxonomy, A2X solves the "Lost-in-the-Middle" problem without sacrificing recall. It allows agents to scale to registries of tens of thousands of services without linear token cost growth.
Autonomy: The system requires no human-in-the-loop ontology maintenance, making it suitable for the dynamic, rapidly evolving "Internet of Agents."
Future Outlook: The authors suggest this approach can serve as a unified discovery layer (an "agent DNS") across heterogeneous services and platforms, enabling richer constraint handling (budgets, exclusions) that vector search cannot easily support.

Limitations Acknowledged:

Static Taxonomy: The current build treats services as static; future work proposes usage-aware refinement to optimize tree depth based on query frequency.
Language Coverage: Cross-lingual results are based on machine-translated data; native non-English registries with incompatible vocabularies may require full rebuilds.
Precision Reporting: Due to incomplete ground-truth annotations in tool-retrieval benchmarks, traditional Precision metrics may under-report the quality of high-recall methods like A2X.