Domain-Adaptive ASR for Telephony AI Agents: Fine-tuning Canary Flash Models for Enterprise Contact Center Applications
This technical report demonstrates that fine-tuning NVIDIA's open-source Canary Flash models on a telephony-specific dataset significantly reduces character error rates for noisy audio and business-critical terminology while maintaining real-time inference speeds for enterprise voicebot deployments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a conversation happening over a telephone line. The sound traveling through that wire is not the same as the clear, high-fidelity voice we hear in a quiet room. It is narrower, often compressed, and frequently interrupted by the static of a busy street or the hum of a cheap handset. For decades, computers have struggled to understand this specific type of audio. While artificial intelligence has become remarkably good at transcribing clear speech from podcasts or meetings, it often stumbles when faced with the messy reality of a customer service call. This gap is a major hurdle for businesses that want to use voice assistants to handle millions of calls, especially in regions where the technology has not been trained on local languages or the specific noise of local phone networks.
Researchers at Botnoi Group in Thailand set out to solve this problem by teaching a powerful, open-source computer model to listen like a human does on a phone call. They focused on the Thai language, a region where public data for telephone speech is scarce. Their work demonstrates that by carefully feeding a model real-world call recordings and business-specific vocabulary, they can transform a general-purpose speech recognizer into a specialist that works reliably in a noisy contact center. The result is a system that not only understands the language but also ignores the static, making it possible for AI agents to handle complex tasks like reading names and addresses without constant human intervention.
The core of their work involved taking a large, pre-existing artificial intelligence model called Canary, which was already good at understanding many languages, and giving it a specialized education. The researchers realized that simply using the model as it was would not work for their needs. The model had been trained mostly on clean, high-quality audio, leaving it unprepared for the distortions of a telephone connection. To fix this, they built a unique training pipeline. They gathered two types of data: recordings from their own live voicebot system, which captured the genuine chaos of real phone calls, and recordings from people speaking prompts into a messaging app. To make the app recordings sound like phone calls, they applied digital filters that mimicked the compression and noise of a telephone line. This process created a massive library of 77 hours of telephone-grade audio, including a specific subset of 104 hours dedicated entirely to the difficult task of recognizing names and addresses.
With this custom dataset, the team fine-tuned the Canary model. Fine-tuning is a process where a computer learns from new examples to adjust its internal rules, much like a student studying a specific textbook to prepare for a particular exam. They tested the model on its ability to recognize Thai speech in three different scenarios. First, they checked if it could understand the language in general. Second, they tested it on the noisy telephone audio. Finally, they tested it on the specific business jargon of names and addresses. The results showed a dramatic improvement. Before this training, the model made errors on telephone audio about 23 percent of the time. After learning from the telephone data, that error rate dropped to just 9 percent. This was a significant leap, proving that the model could adapt to the specific acoustic conditions of a phone line.
The challenge of names and addresses is particularly difficult for computers because these words are often unique and can sound very similar to one another. In the initial tests, the model struggled with these terms, making errors nearly 17 percent of the time. However, after the researchers gave the model a second round of training focused specifically on the names and addresses dataset, the error rate plummeted to less than 4 percent. This level of accuracy is critical for enterprise applications, where a mistake in a customer's name or address can lead to a failed transaction or a frustrated user. The study also compared two versions of the model: a smaller, faster version and a larger, more complex one. While the larger model was slightly more accurate, the smaller version was nearly as good but ran much faster, making it the better choice for real-time systems that need to respond instantly.
Throughout the experiments, the researchers measured how quickly the computer could process the audio. They found that the smaller model could handle the workload with incredible speed, processing audio more than 600 times faster than real-time on standard hardware. This means that even a single computer chip could handle thousands of calls simultaneously without lag. The study confirms that by combining a powerful, pre-trained model with a carefully collected, domain-specific dataset, it is possible to build robust voice systems for languages and environments that have previously been underserved. The work does not claim to have solved every problem in speech recognition, nor does it suggest that the system is perfect in every possible situation. Instead, it offers a clear, proven path for adapting general AI tools to the specific, noisy, and high-stakes world of enterprise telephony, showing that with the right data, machines can learn to listen just as well as humans do on the phone.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.