💻 computer science

Semantic Tool Discovery for Large Language Models: A Vector-Based Approach to MCP Tool Selection

This paper introduces a vector-based semantic tool discovery architecture for the Model Context Protocol (MCP) that significantly reduces token overhead and latency by dynamically selecting only the most relevant tools via dense embeddings, achieving a 99.6% reduction in token consumption while maintaining a 97.1% hit rate.

Original authors: Sarat Mudunuri, Jian Wan, Ally Qin, Srinivasan Manoharan

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sarat Mudunuri, Jian Wan, Ally Qin, Srinivasan Manoharan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant but slightly overwhelmed personal assistant (the Large Language Model or LLM). You are incredibly smart at writing, coding, and solving problems, but you can't physically pick up a phone, check a database, or look at a file on your own. You need tools to do those things.

In the past, to help you, your boss (the system) would hand you a giant, dusty encyclopedia containing every single tool they own—maybe 100, 200, or even 1,000 different ones. They would say, "Here is the whole list. Figure out which one you need for this job."

The Problem: The "Library of Everything"

This approach has three big flaws:

It's too expensive: Reading the descriptions of 1,000 tools takes up a massive amount of "mental space" (tokens), which costs real money.
It's confusing: When you are staring at 1,000 options, you get "analysis paralysis." You might pick the wrong one because you're overwhelmed.
It's slow: You spend more time reading the list than actually doing the work.

The Solution: The "Smart Librarian"

This paper proposes a new way to work. Instead of handing you the whole encyclopedia, they install a Smart Librarian (the Vector-Based Discovery System) between you and the tools.

Here is how the new system works, step-by-step:

1. The Tool Index (The Library Catalog)

First, the system takes all those 100+ tools and writes a short, friendly summary for each one. It turns these summaries into a unique "fingerprint" (a vector embedding) that captures the meaning of the tool, not just the words.

Analogy: Imagine every tool is a book. Instead of just looking at the title, the librarian reads the whole book and gives it a specific color code based on its vibe. "File Deleter" gets a "Red" code; "Weather Checker" gets a "Blue" code.

2. The User Query (Asking for Help)

When you ask a question like, "I need to delete that old report," the system doesn't just look for the word "delete." It turns your sentence into a matching "fingerprint."

Analogy: You tell the librarian, "I need a Red book." The librarian instantly knows what you mean, even if you said "get rid of" instead of "delete."

3. The Magic Match (Vector Search)

The librarian looks at their color-coded list and instantly finds the 3 to 5 tools that match your fingerprint best.

The Result: Instead of handing you 100 books, the librarian hands you a tiny stack of 3 perfectly relevant books.

Why This is a Game-Changer

The paper tested this with 121 different tools (like checking files, looking up weather, or managing GitHub projects) and 140 different questions. Here is what they found:

Massive Savings (The Wallet): By only showing the LLM 3 tools instead of 121, they cut the "reading cost" by 99.6%.
- Real-world impact: If a company was spending $1.5 million a year on "reading the tool list," this method drops that cost to almost nothing.
Smarter Choices (The Brain): The LLM made fewer mistakes. When you give a genius a tiny, perfect list, they perform better than when you give them a giant, messy list. The system found the right tool 97% of the time just by showing the top 3 options.
Super Fast (The Speed): The librarian did the matching in less than 100 milliseconds. That's faster than a human blinking. It adds almost no delay to the conversation.

The "File System" Challenge

The paper also noted that some tools are harder to distinguish than others.

Easy: "Check the weather" vs. "Check GitHub." These are very different. The librarian finds them instantly.
Hard: "Read a file," "Write a file," "Copy a file," "Move a file." These all sound very similar. The librarian sometimes has to show a few more options (like 5 instead of 3) to make sure the LLM picks the right one.

The Bottom Line

This paper introduces a filter for AI tools. Instead of drowning the AI in a sea of options, it acts as a smart gatekeeper, only letting the most relevant tools through the door.

In simple terms: It turns a chaotic, expensive, and confusing "all-you-can-eat buffet" of tools into a curated, 3-course meal that is cheaper, faster, and tastes much better. This makes it possible for companies to use AI with hundreds of tools without going broke or getting confused.

1. Problem Statement

The paper addresses a critical scalability bottleneck in Model Context Protocol (MCP) ecosystems. As Large Language Models (LLMs) integrate with external tools via MCP, production systems often connect to multiple servers exposing hundreds of tools (e.g., 50–100+ tools).

Current approaches face four major limitations when providing the full tool catalog to the LLM:

Token Overhead: Tool schemas (JSON) consume 200–800 tokens per tool. A catalog of 100 tools can consume 20,000–80,000 tokens before the user query is even processed.
Cost: High token consumption leads to prohibitive costs (e.g., ~$1.50 per request at GPT-4 pricing, scaling to millions monthly).
Accuracy Degradation: "Cognitive overload" occurs when LLMs are presented with irrelevant tools, reducing selection accuracy.
Context Window Constraints: Large catalogs compete with conversation history and retrieved documents for limited context space.

Existing solutions (static provisioning or manual categorization) lack the flexibility to dynamically match user intent with specific tools based on semantic nuance.

2. Methodology

The authors propose a Semantic Tool Discovery Architecture that replaces static tool provisioning with a dynamic, vector-based retrieval system.

System Architecture

The system operates in four stages:

Tool Indexing Pipeline:
- Discovery: Queries MCP servers to enumerate tools.
- Document Construction: Creates rich text representations for each tool, including the name, purpose, expanded capabilities, and parameter descriptions.
- Embedding: Converts these text representations into dense vectors using an embedding model (tested with text-embedding-ada-002).
- Storage: Indexes vectors with metadata in a vector database (Milvus).
Query Processing:
- The user query is embedded into the same vector space.
- A similarity search retrieves the top- $K$ most relevant tools.
- Optional filtering (similarity thresholds) and reranking can be applied.
LLM Integration:
- Only the top- $K$ selected tools are formatted and injected into the LLM context alongside the user query.
- The LLM executes tool calls, and results are fed back for response generation.
Feedback Loop:
- The system optionally logs actual tool invocations and success rates to refine embeddings or adjust retrieval parameters.

Key Technical Parameters

Embedding Model: text-embedding-ada-002 (1536 dimensions).
Similarity Metric: Dot Product.
Retrieval Strategy: Top- $K$ selection (tested $K \in \{1, 2, 3, 5, 10\}$ ).
Implementation Stack: Python, Milvus Vector Store, Official MCP SDK, GPT-4o.

3. Key Contributions

Novel Architecture: The first semantic tool discovery layer specifically designed for the MCP ecosystem, utilizing vector embeddings for dynamic selection.
Comprehensive Evaluation: Extensive experiments across 5 diverse MCP servers (Filesystem, MySQL, Slack, GitHub, Time/Weather) and 121 tools.
Open-Source Implementation: A production-ready, open-source framework designed for seamless integration.
Benchmark Dataset: Release of a curated dataset containing 140 queries with ground-truth tool selections across 5 servers.
Extensibility Analysis: Demonstrates applicability to multi-agent routing, cross-organizational discovery, and dynamic tool composition.

4. Experimental Results

The system was evaluated on 140 queries across 121 tools. The optimal operating point was found to be $K=3$ .

Performance Metrics (at $K=3$ )

Hit Rate: 97.1% (The correct tool is present in the top 3 results).
Mean Reciprocal Rank (MRR): 0.91 (The correct tool is highly likely to be the very first result).
Token Reduction: 99.6% (Drastic reduction in context tokens compared to providing all 121 tools).
Latency: < 100ms (Specifically ~87.8ms), adding negligible overhead to the pipeline.
F1 Score: 58.4% (Balancing precision and recall).

Domain-Specific Insights

High Performance: Servers with distinct tool descriptions (GitHub, MySQL) achieved near-perfect hit rates at low $K$ values.
Challenging Domains: The Filesystem server showed lower performance due to high semantic overlap between tools (e.g., read_file vs. write_file), requiring higher $K$ for adequate recall.
Small Catalogs: The Time/Weather server (10 tools) showed high F1 scores because a larger fraction of its small catalog is relevant to any given query.

5. Significance and Implications

Economic Viability: By reducing token consumption by 99.6%, the approach transforms large-scale MCP deployments from economically prohibitive to viable, potentially saving millions in operational costs.
Improved Accuracy: Counter-intuitively, providing fewer (but more relevant) tools improves LLM accuracy. Reducing the tool set from 100+ to 3–5 minimizes noise, allowing the LLM to focus reasoning capacity on parameter construction rather than tool disambiguation.
Scalability: The architecture solves the "context window" bottleneck, enabling LLMs to access thousands of tools without exceeding context limits.
Future Directions: The paper outlines how this vector-based approach can evolve into multi-agent routing (selecting the right agent, then the right tool) and dynamic tool composition (retrieving entire tool chains/workflows).

Conclusion

This paper demonstrates that semantic vector retrieval is a superior alternative to static tool provisioning for LLMs. It achieves a 99.6% reduction in token costs while maintaining a 97.1% hit rate for tool selection, effectively solving the scalability challenge of the Model Context Protocol and paving the way for robust, production-grade agentic AI systems.