🤖 AI

AgentSearchBench: A Benchmark for AI Agent Search in the Wild

AgentSearchBench is a large-scale benchmark designed to evaluate agent discovery by formalizing agent search as a retrieval and reranking problem, demonstrating that traditional semantic similarity fails to predict actual performance and that incorporating execution-based signals is essential for finding suitable agents.

Original authors: Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine Yilmaz

Published 2026-04-27

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine Yilmaz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in the middle of a massive, infinite shopping mall. This mall doesn't just sell clothes or electronics; it sells "Digital Workers" (AI Agents).

Some workers are experts at coding, some are master chefs, and others are professional travel planners. The problem? There are nearly 10,000 of them, and they all have confusing brochures. One worker’s brochure says, "I am a master of logic," while another says, "I can solve complex math problems." You want someone to "Organize a 3-day trip to Tokyo," but you have no idea which worker can actually do it versus which one just sounds good on paper.

This paper introduces AgentSearchBench, a high-tech "Testing Lab" designed to solve this exact problem.

The Core Problem: The "Resume vs. Reality" Gap

In the world of AI, there is a massive gap between what an agent says it can do (its description) and what it actually does (its performance).

Think of it like hiring a personal trainer based on their Instagram profile.

The Description (Semantic Similarity): You see a photo of someone in gym gear with a caption about "strength training." You think, "Perfect, they can help me lift weights!"
The Reality (Execution): You show up to the gym, and it turns out they only know how to do yoga.

Traditional search engines look for keywords. If you search for "fitness," they find anyone who mentions "fitness." But in the AI world, matching the words isn't enough; you need to match the capability.

What is AgentSearchBench?

The researchers built a massive playground to test how well search engines can find the right AI workers. Here is how they did it:

The Massive Talent Pool: They gathered nearly 10,000 real-world AI agents from actual platforms (like the GPT Store).
The "Stress Test" (Execution-Grounded): Instead of just checking if the agent's description matches your request, they actually hire the agent and make them work. They give the agent a task, watch it try to complete it, and grade it on how well it actually performed.
Two Levels of Difficulty:
- The Clear Order (Task Queries): "Write a Python script to scrape this website." (Very specific).
- The Vague Dream (Task Descriptions): "I want to modernize my business workflow." (Very broad and fuzzy).

The Big Discovery: "Don't Trust the Brochure"

The researchers found that most current search methods are actually quite bad at this. They rely too much on "surface matching"—looking for similar words.

Their experiments showed that an agent might have a perfect description for your task but fail miserably when actually doing it. Conversely, an agent might have a weird, poorly written description but turn out to be a superstar performer.

The Solution: "The Trial Run" (Probing)

The paper suggests a better way to find the right agent: Execution-Aware Probing.

Instead of just reading the brochure, imagine if, before hiring a chef, you asked them to "quickly scramble an egg" just to see if they actually know how to use a stove. This "lightweight trial run" provides a behavioral signal.

The researchers proved that if you give the search engine a tiny bit of "real-world evidence" (seeing how the agent reacts to a small test), the quality of the search improves significantly.

Summary in a Nutshell

AgentSearchBench is a way to stop being fooled by "fancy resumes" in the AI world. It teaches us that to find the best AI assistants, we shouldn't just look for the ones that talk the best; we need to look for the ones that work the best.

Technical Summary: AgentSearchBench

AgentSearchBench is a large-scale benchmark designed to address the challenges of identifying and selecting suitable AI agents within open, real-world ecosystems. Unlike traditional tool retrieval, which focuses on static, well-defined APIs, agent search must account for the compositional, execution-dependent, and often inconsistently documented nature of autonomous agents.

1. The Problem: The Semantic–Performance Gap

The authors identify a fundamental misalignment in current agentic systems: semantic similarity does not equal functional competence.

Compositionality: Agent capabilities are often emergent and depend on how they interact with tools, making them hard to capture via text alone.
Documentation Inconsistency: In "in the wild" ecosystems (like the GPT Store), agent descriptions are non-uniform and often fail to reflect actual performance.
Query Complexity: Real-world users often provide high-level, non-executable task descriptions rather than precise, structured instructions.
Existing Limitations: Current benchmarks focus on small, controlled candidate pools or assume that functionality can be inferred solely from structured schemas.

2. Methodology

The researchers propose a hierarchical pipeline to build a realistic and execution-grounded benchmark.

A. Data Collection (AgentBase):
They constructed a repository of nearly 10,000 real-world agents sourced from diverse platforms (GPT Store, Google Cloud Marketplace, AgentAI). This ensures the benchmark captures practical challenges like overlapping capabilities and messy documentation.

B. Task Generation Pipeline:

Executable Task Queries ( $T_q$ ): Synthesized from agent documentation to create concrete, runnable instructions (both single-agent and multi-agent).
High-Level Task Descriptions ( $T_d$ ): Generated by abstracting clusters of related executable queries into high-level goals. Each description is linked to a set of queries ( $Q(T_d)$ ) to evaluate consistent capability.

C. Execution-Grounded Evaluation:
Relevance is not determined by text matching but by task performance.

Agents are executed on queries.
An LLM-as-a-judge evaluates the quality of the output on a 5-point scale.
Relevance Labels: For retrieval, a binary threshold is used; for ranking, a graded scale is used based on subtask completion.
Reliability Weighting: To account for "lucky" successes, agents that solve tasks without documented capability are assigned discounted relevance scores.

3. Key Contributions

New Problem Formulation: Formalizes agent search as a retrieval and reranking problem under execution-dependent capability uncertainty.
Large-Scale Benchmark: Provides a massive, diverse dataset of ~10k agents and ~3k tasks.
Execution-Grounded Framework: Shifts the evaluation paradigm from "does the description match the query?" to "does the agent actually solve the task?"
Behavioral Probing Discovery: Introduces the concept of using lightweight execution signals (probing) to improve ranking.

4. Key Results

The Semantic–Performance Gap: Experiments reveal that description-based retrieval (e.g., BM25, Dense embeddings) often fails to surface the highest-performing agents. High-performing agents are frequently ranked lower because their actual competence exceeds their textual documentation.
Retrieval vs. Reranking:
- Retrieval: Tool-aware retrievers (like ToolRet) perform best on concrete queries, while dense retrievers (like BGE) are more competitive for abstract task descriptions.
- Reranking: Decoder-only and LLM-based rerankers (e.g., RankGPT) perform significantly better on high-level descriptions, likely due to their ability to infer implicit requirements.
Effectiveness of Probing: The study demonstrates that Execution-Aware Probing—running a few lightweight queries to observe agent behavior—substantially improves ranking quality (NDCG). This proves that behavioral signals are a vital complement to static text.

5. Significance

AgentSearchBench provides the first realistic testbed for the "discovery" phase of the agentic workflow. As the ecosystem of third-party agents grows, the ability to reliably find the right agent for a complex, multi-step task becomes a bottleneck for both end-users and automated orchestration systems. This work establishes a new standard for evaluating agentic systems by prioritizing demonstrated capability over textual promise.