SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise
This paper introduces SQuTR, a large-scale robustness benchmark for spoken query to text retrieval that aggregates diverse queries and synthesizes speech under 17 categories of real-world noise to evaluate and diagnose the performance limitations of retrieval systems in complex acoustic environments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to find a specific recipe in a giant library, but instead of typing your request, you shout it out loud. This is the world of Spoken Query Retrieval, a technology that powers voice assistants in our cars, homes, and pockets. For these systems to work, they need two superpowers working in tandem: first, a "listener" (called Automatic Speech Recognition, or ASR) that turns your voice into text, and second, a "searcher" (Information Retrieval) that finds the right documents based on that text. In a quiet room, this dance works beautifully. But in the real world, life is messy. Background noise, echoey hallways, and chatty crowds can turn your clear request into a garbled mess. If the listener mishears "find me a pizza place" as "find me a piss place," the searcher will get confused and fail. Scientists have long known that noise is a problem, but they've mostly tested these systems in sterile, quiet labs or with very simple, short questions. They haven't really checked how well these systems handle the chaotic, noisy reality of a busy subway station or a windy park.
Enter SQuTR, a new "stress test" for spoken search systems, created by a team of researchers to see how well these digital librarians hold up when the world gets loud. Think of SQuTR as a giant, noisy obstacle course designed specifically for voice search. The researchers didn't just make up fake questions; they took over 37,000 real queries from six different existing search datasets, covering everything from finance and medicine to general trivia and complex multi-step questions. They then used a computer voice generator to turn these text questions into speech, using 200 different real human voices to ensure variety.
Here is where the experiment gets really interesting. The team didn't just play the audio in a quiet room. They took their digital voices and dropped them into a blender of 17 different real-world noise types—like the roar of a bus, the chatter of a cafeteria, or the hum of an office. They created four distinct levels of chaos: Clean (silence), Low Noise (20 dB, like a quiet conversation nearby), Medium Noise (10 dB, like a busy street), and High Noise (0 dB, where the noise is just as loud as the voice). This setup allowed them to measure exactly how much the search performance drops as the noise gets louder, from a whisper to a shout.
The researchers tested two main types of search systems. The first is the Cascaded System, which is like a relay race: the voice is passed to a transcriber to turn it into text, and then that text is handed to a search engine. The second is the End-to-End System, which tries to skip the transcription step entirely, mapping the raw sound directly to search results. They ran these systems against a massive library of documents and measured how well they found the right answers using standard search scores.
The results were a clear, loud warning. As the noise levels increased, retrieval performance dropped significantly across the board. Even the most advanced, large-scale models struggled when the noise got extreme. The paper found that while larger models (like the 8-billion-parameter Qwen3-Embedding) held up better than smaller ones, they still couldn't match the performance of a clean text search, even in the "Clean" audio condition. This suggests that the gap between hearing a voice and understanding it perfectly is a massive bottleneck that current technology hasn't solved yet.
Perhaps the most surprising finding was about the "relay race" versus the "direct map." The study showed that simply making the voice transcriber better (by using a bigger, smarter ASR model) didn't fix the problem as much as you might think. Instead, the type of search engine used mattered more. Systems that used dense retrieval (which understands the meaning of words) were much more robust against noise than those using lexical retrieval (which just matches exact keywords). In fact, a smaller voice transcriber paired with a smart, meaning-based search engine often outperformed a giant voice transcriber paired with a simple keyword matcher. This suggests that if we want voice search to work in a noisy world, we need to focus less on perfecting the transcription and more on building search engines that can understand the intent behind the garbled words.
Ultimately, SQuTR doesn't offer a magic fix, but it provides a crucial, reproducible map of the problem. It proves that noise is a critical, unsolved challenge for spoken search and shows that the path forward lies in combining better noise-tolerant search strategies with controlled, realistic testing. The researchers have made their dataset and code public, inviting the rest of the scientific community to join the race to build voice assistants that can hear us clearly, even when the world is screaming.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.