BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models

This paper introduces a benchmark to reveal significant tool-selection bias in large language models driven by metadata alignment and pre-training exposure, and proposes a lightweight filtering-and-sampling strategy to mitigate these fairness issues while maintaining task coverage.

The Gist

🍽️ The Problem: The "First on the Menu" Effect

Imagine you walk into a massive restaurant called "The LLM Diner." You are hungry and ask the waiter (the AI) for a burger.

In a perfect world, the waiter would look at the 50 different burger options available, taste-test them (metaphorically), and pick the one that tastes best or is the freshest.

But in reality, the waiter has a weird habit. They almost always pick the burger from the first stall on the menu, or the one with the fanciest name written in bold letters, even if the burger next to it is actually better and cheaper.

This paper calls this "Tool-Selection Bias."

Large Language Models (LLMs) are becoming like these waiters. They are being taught to use external tools (like weather apps, translation services, or stock checkers) to do their jobs. But when there are five different weather apps that all do the exact same thing, the AI doesn't pick randomly. It has a favorite. It might pick the one that appears first in the list, or the one with a name that sounds "cooler," ignoring the fact that the other options are just as good.

🕵️‍♂️ The Investigation: Why Does the AI Do This?

The researchers built a "test kitchen" (a benchmark) to figure out why the AI is being so picky. They created groups of tools that were functionally identical (like 5 different brands of identical umbrellas) and asked the AI to pick one.

Here is what they discovered:

1. The "Name Game" is powerful: If a tool has a name that sounds very similar to what you asked for, the AI picks it. It's like the waiter picking the "Super Burger" over the "Great Burger" just because the word "Super" is in the name, even if they are the same burger.
2. The "First Seat" rule: If you list the tools in a specific order, the AI loves the one at the very top. It's like a student raising their hand first and getting called on, even if the student in the back row has the better answer.
3. The "Training Memory" effect: If the AI was trained on a lot of text that mentioned one specific tool (like a specific weather app) over and over again, it will keep picking that one, even if it's not the best choice. It's like a waiter who only knows one brand of ketchup because their boss only ever bought that brand.

📉 Why Should You Care? (The Consequences)

You might think, "So what? The AI picked a weather app. It's still going to tell me if it's raining."

But here is why it matters:

• The Unfair Business: Imagine the "Super Burger" stall is owned by a big corporation, and the "Great Burger" stall is owned by a small local family. If the waiter always picks the big corporation's burger just because of the name, the small family goes out of business. This creates an unfair market where only the "famous" tools survive, and innovation dies.
• The Slow & Expensive Route: Sometimes the AI picks a tool that is slow or expensive just because it was listed first. This makes the AI slower for you and costs more money for the company running it.
• The "Hacker" Risk: If the AI is easily tricked by a fancy name, a bad actor could rename a malicious tool to sound safe and get the AI to use it.

🛠️ The Solution: The "Fair Filter"

The researchers didn't just point out the problem; they built a fix. They call it BiasBusters.

Think of it like a smart bouncer at the restaurant door.

1. Step 1 (The Filter): Before the waiter (the main AI) sees the list of 50 burgers, the bouncer (a smaller, simpler AI) looks at the list. The bouncer says, "Okay, the customer wants a burger. These 5 options are all good burgers. These other 45 are salads or desserts. Let's throw the salads away."
2. Step 2 (The Random Pick): Now, the waiter only has to choose from those 5 good burgers. The bouncer tells the waiter: "Pick one of these 5 completely at random."

The Result:
By filtering out the irrelevant options first and then forcing a random choice among the good ones, the bias disappears. The AI stops favoring the "fancy name" or the "top of the list" and starts treating all the valid tools fairly.

🎯 The Big Takeaway

This paper is a wake-up call. As we build more "AI Agents" that can go out and do things for us (like booking flights or checking stocks), we need to make sure they aren't secretly rigged to favor certain companies or tools just because of how they are named or ordered.

BiasBusters shows us that with a little bit of smart filtering, we can make AI agents fairer, cheaper, and more reliable for everyone. It's about making sure the AI picks the right tool, not just the loudest one.

Technical Summary

1. Problem Statement

Large Language Model (LLM) agents increasingly rely on external tools (APIs) from marketplaces (e.g., RapidAPI) where multiple providers offer functionally equivalent services. The paper identifies a critical tool-selection bias: LLMs systematically favor specific providers over others not based on utility or relevance, but due to superficial factors such as:

• Metadata: Tool names, descriptions, and parameter schemas.
• Positional Bias: The order in which tools appear in the prompt.
• Pre-training Exposure: Frequency of exposure to specific endpoints during training.

This bias leads to:

• Market Unfairness: Disproportionate revenue for favored providers and loss for competitors offering identical functionality.
• Degraded User Experience: Selection of slower, less reliable, or more expensive tools.
• Operational Costs: Inflated costs due to inefficient routing.

2. Methodology

A. Benchmark Construction (BiasBusters)

The authors created a novel benchmark to systematically evaluate tool-selection bias:

• Data Source: APIs scraped from RapidAPI.
• Clustering: APIs were clustered into 10 groups based on functional equivalence (e.g., weather forecasting, geocoding, email validation). Each cluster contains 5 functionally identical APIs.
• Query Generation: 1,000 user queries were generated (100 per cluster) that are provider-agnostic and can be satisfied by any API in the cluster.
• Evaluation Protocol:
  • Models are prompted to select one tool from the list.
  • Cyclic Rotation: To control for positional bias, each query is run 5 times with the API list cyclically rotated, ensuring every API appears at the top position exactly once.

B. Bias Metrics

The paper introduces Total Variation (TV) Distance metrics to quantify bias:

• $\delta_{API}$: Measures deviation from a uniform distribution across APIs within a cluster (API-level bias).
• $\delta_{pos}$: Measures deviation from a uniform distribution across list positions (positional bias).
• $\delta_{model}$: The average of $\delta_{API}$ and $\delta_{pos}$, representing overall model bias.

C. Analysis of Drivers

To explain the sources of bias, three complementary analyses were conducted:

1. Attribute-Level Analysis: Correlation and regression analysis between API features (e.g., description length, semantic similarity to query, readability) and selection rates.
2. Metadata Perturbation Experiments: Controlled modifications to tool metadata to isolate causal factors:
   • Name Scrambling/Shuffling: Testing reliance on literal names.
   • Description/Parameter Scrambling: Testing reliance on semantic content.
   • Targeted Swaps: Swapping descriptions between the most and least selected tools.
3. Biased Continued Pre-training (CPT): Fine-tuning a model (Qwen3-8B) on a corpus saturated with metadata from a single target endpoint to test if exposure alone induces preference.

D. Mitigation Strategy

A lightweight Debiasing Module was proposed:

1. Filtering: A smaller LLM (Qwen3-14B) acts as a "subset selector," identifying all APIs in the candidate list capable of solving the user's query.
2. Uniform Sampling: The final tool is chosen uniformly at random from this filtered subset, eliminating positional and metadata-based favoritism while maintaining task coverage.

3. Key Results

A. Existence and Magnitude of Bias

• Universal Bias: All 7 evaluated models (GPT-3.5, GPT-4.1-mini, Claude 3.5, Gemini 2.5, DeepSeek, ToolLLaMA, Qwen3) exhibit substantial bias.
• Metrics: Combined bias ($\delta_{model}$) ranges from 0.25 to 0.38. This implies that roughly 30–40% of selection probability mass would need to be redistributed to achieve fairness.
• Patterns:
  • Some models fixate on a single provider (high $\delta_{API}$).
  • Others rely heavily on list position (high $\delta_{pos}$).
  • Models show high alignment in their bias patterns (e.g., GPT-4.1, Claude, and Gemini favor/disfavor the same APIs), suggesting shared implicit decision rules.

B. Drivers of Bias

1. Semantic Alignment is Key: The strongest predictor of selection is the semantic similarity between the user query and the tool's description. Structural features (parameter count, age) had little influence.
2. Metadata Sensitivity:
   • Perturbing descriptions caused the largest shifts in selection behavior (Mean TV distance ~0.45 for Gemini).
   • Name-only perturbations had smaller, noisier effects.
   • Targeted Swaps: Swapping the description of the most-selected tool with the least-selected one often inverted their selection rates, proving descriptions drive choice more than names.
3. Pre-training Exposure: Biased CPT on a single endpoint increased its selection share from 0.6% to ~12.8% after one epoch. However, this did not fully dominate selection, indicating pre-training is a factor but not the sole driver.
4. Model Size: Larger models (e.g., Qwen3-235B) exhibited slightly less bias than smaller counterparts.

C. Mitigation Effectiveness

• The subset-selection approach successfully reduced bias metrics:
  • $\delta_{API}$ dropped from 0.338 to 0.108.
  • $\delta_{pos}$ dropped from 0.422 to 0.079.
• Performance: The subset selector achieved ~99.6% precision (rarely including invalid tools) and ~88.6% recall (retaining most valid tools), ensuring task solvability was not compromised.

4. Key Contributions

1. First Empirical Study: The first large-scale benchmark and systematic study quantifying tool-selection bias in LLM agents.
2. Root Cause Analysis: Identification that semantic alignment in metadata and pre-training exposure are primary drivers, with description content being more influential than tool names.
3. Mitigation Framework: A practical, lightweight, and effective strategy (filter + uniform sampling) that significantly reduces bias without requiring model retraining.
4. Open Resources: Public release of the benchmark, code, and evaluation pipeline to facilitate future research.

5. Significance and Implications

• Economic Fairness: The study highlights how LLM bias can distort the API marketplace economy, creating winners and losers based on metadata phrasing rather than technical merit.
• System Reliability: Biased selection increases vulnerability to adversarial attacks (e.g., metadata poisoning) and degrades user experience by routing traffic to suboptimal services.
• Future of Agentic AI: As LLM agents become ubiquitous, ensuring fair and unbiased tool selection is critical for the trustworthiness and stability of the broader AI ecosystem. The proposed mitigation offers an immediate, deployable solution for practitioners.

Conclusion: The paper establishes that tool-selection bias is a pervasive, reproducible phenomenon in current LLMs. It is driven largely by semantic cues in metadata and training exposure, but can be effectively mitigated through a simple filtering-and-sampling architecture, paving the way for fairer and more reliable agentic systems.