When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval
This paper identifies "source-style collapse," a failure mode where tool-backed retrievers generalize poorly across different data sources despite high lexical overlap, and proposes ToolScout, a TF-IDF-based routing method that significantly improves retrieval coverage by detecting and guarding against these style mismatches.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a digital assistant that can perform complex tasks by calling upon a vast library of external tools, much like a human chef reaching for specific ingredients from a massive pantry. To function, this assistant must first locate the right tool from thousands of options before it can even begin to work. This initial search is a critical gatekeeper; if the assistant fails to find the correct tool in its first attempt, the entire task fails, because no amount of clever planning later can recover a missing ingredient. For years, researchers have focused on making these assistants smarter at planning and reasoning, assuming that the search for the right tool was a solved problem. However, a new study reveals a hidden flaw in how these systems search: the search engine itself can suddenly stop working, not because the tools have disappeared, but because the way people ask for them has changed.
The researchers investigated a phenomenon they call "source-style collapse." In the world of artificial intelligence, tools are often organized into large collections, and the system is trained to find the right one based on how a user describes their need. The team discovered that a search system trained to understand one specific style of language could become completely useless when faced with a different style, even if the tools themselves remained exactly the same. To test this, they used a system trained on a set of 1,100 queries written in a specific, structured format. When they tested this same system on a different set of queries generated by a different source, the system's ability to find the correct tool plummeted. In one striking case, the system found the right tool less than one percent of the time, despite the new queries using words that were actually more similar to the correct tools than the original training data. This failure happened silently and catastrophically, leaving the assistant blind to the very capabilities it needed.
The study ruled out several obvious explanations for this breakdown. It was not a problem with the length of the questions, nor was it simply a matter of vocabulary overlap. The researchers also tested whether the failure was caused by the raw technical format of the tool descriptions, which often look like complex code. They re-wrote every tool into a plain, readable "skill card" that described what the tool did, when to use it, and what inputs it needed. Even with these human-friendly descriptions, the search system still collapsed when the style of the question changed. This proved that the issue was not the look of the tools, but a deeper mismatch between the way the system learned to search and the way the new questions were phrased. The system had become too specialized in one dialect of language to understand another, even when both dialects were asking for the exact same thing.
To solve this, the team developed a new method called ToolScout, which acts as a traffic director for the search process. Instead of forcing every question through a single, specialized search engine, this new system first checks the "fingerprint" of the incoming question. It uses a simple, lightweight analysis of the words to determine if the question matches the style the search engine was trained on. If the question looks like it belongs to a different style, the system immediately routes it to a broader, more general search engine that has seen many different styles of questions. This switch happens instantly and automatically. When they applied this routing method to a mixed stream of nearly 5,000 questions, the system's ability to find the right tool jumped from a failing 22 percent to a reliable 86 percent.
The researchers also found that once the system was routed to the correct engine, it could be quickly improved with just a tiny amount of new information. By showing the specialized search engine just 20 examples of the new style of questions, they could repair its performance, raising its success rate from near zero to over 50 percent for those specific cases. This suggests that the problem is not a fundamental flaw in the technology, but a lack of flexibility in how the search engine adapts to new ways of speaking. The study concludes that for digital assistants to be truly reliable, they cannot just be smart planners; they must also have a robust, adaptable way of finding the right tools, one that recognizes when the language of the user has shifted and knows how to switch gears before the task even begins.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.