INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval

Imagine trying to find a specific needle in a haystack, but the needle changes its shape depending on how you ask for it. If you ask for a needle that is shiny, you get a different one than if you ask for a needle that is red, or one that was dropped near a loud noise. For decades, computers have been getting better at finding information, but they have mostly relied on a rigid system of matching. If you search for a word, the computer looks for that exact word. If you search for a sound, it looks for a sound that sounds exactly like your search. This works well when you know exactly what you are looking for, but it fails when your needs are more complex. In the real world, people do not just want to find a recording that sounds the same; they might want to find a recording where the speaker sounds angry, or where the background noise is the sound of rain, or where the person speaking is the same one who spoke in a previous clip. The challenge for scientists has been to teach computers to understand these shifting, human-like instructions rather than just matching static patterns.

A team of researchers at National Taiwan University has taken a major step toward solving this problem by creating a new testing ground called INSPIRE. This is not a new computer program that fixes everything, but rather a carefully constructed benchmark designed to see how well current technology can handle these flexible instructions. The researchers built a massive library of spoken recordings and paired them with thousands of natural language commands. These commands range from simple requests, like "find the next part of this conversation," to complex, multi-layered demands, such as "find a recording of the same person speaking sadly while footsteps are heard in the background." The goal was to see if existing artificial intelligence systems could look at a spoken question and a written instruction, then successfully hunt down the exact audio clip that fits that specific description.

The researchers tested four different types of computer systems to see how they fared. The first group consisted of large audio-language models, which are advanced systems trained to understand both sound and text. The second group used a "cascaded pipeline," a method where the computer first turns the spoken audio into written text and captions, and then searches for that text using standard text-search tools. The third group relied on self-supervised speech models, which learn to understand sound patterns directly without converting them to text. The final group used contrastive models that try to link text and audio together. The team measured how often each system successfully found the correct audio clip out of thousands of wrong options.

The results revealed a clear and surprising split in how these machines think. The systems that convert speech into text first were excellent at finding clips based on what was actually said. If the instruction was to find a specific sentence or a continuation of a conversation, these text-based methods performed very well. However, they struggled immensely when the instruction was about how the sound was made. When asked to find a clip based on the speaker's identity, their emotional tone, or the background noise, these text-based systems performed no better than random guessing. They simply could not "hear" the difference between a happy voice and a sad one once the sound was turned into words.

On the other hand, the systems that listened directly to the sound waves without turning them into text showed the opposite strength. They were much better at recognizing the speaker's voice, the style of their speech, and the environmental sounds around them. But these same systems failed when the instruction required them to understand the meaning of the words or to follow a complex, multi-part command. They could hear the voice, but they could not follow the instruction to find a specific voice speaking a specific thing in a specific way. Even the most advanced large audio-language models, which were expected to bridge this gap, could not do it. They performed reasonably well on some tasks but fell short when the task required a deep understanding of both the sound and the instruction simultaneously.

The study also looked at what happens when the computer is given a "reference" of perfect metadata, such as knowing exactly who the speaker is or what the background noise is, rather than trying to figure it out from the audio. Even with this perfect information, the text-based systems still could not reliably find the right clips when the instructions involved complex combinations of features. This suggests that the problem is not just that the computers are bad at hearing, but that their current architecture is not built to handle instructions that mix different types of information. The researchers found that no single method they tested could robustly handle all the different ways a human might ask to find a sound.

Ultimately, the paper concludes that the field is at a crossroads. The current tools are too specialized; some are great at reading the transcript but deaf to the tone, while others are great at hearing the tone but illiterate in the instructions. The researchers argue that the next generation of technology needs to be a unified system, one that can listen to a sound, read a complex instruction, and understand that the answer lies in the combination of both. Until such a system is built, the ability to search a library of voices with the same natural ease and precision that we search for text remains out of reach. The work does not offer a final solution, but it clearly maps the terrain, showing exactly where the current technology breaks down and where the future of speech retrieval must go.