Benchmarking Speech Recognition Models for Medical Consultations in Latin American Spanish: A Comparative Evaluation with Fine-Tuning
This study benchmarks ten speech-to-text models on Latin American Spanish medical consultations, finding that while fine-tuning Whisper Large v3 did not surpass its vanilla version, it outperformed other open-source and closed-source models on key metrics, establishing it as the optimal open-source solution for AI medical scribes in this context.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Imagine a world where your doctor doesn't have to type a single word while you talk. Instead, a smart computer listens to your entire visit, writes down exactly what you said, and turns it into a neat medical note. This is the dream of the "AI medical scribe." But for this to work, the computer needs to be incredibly good at one specific job: listening to human speech and turning it into text. This job is called Speech-to-Text (STT). You might know it from your phone's voice assistant, but those assistants are usually trained on clear, standard English. The real world, however, is messy. People speak with different accents, use slang, talk over each other, and use complicated medical words. In Latin America, where Spanish is spoken with a huge variety of regional flavors and dialects, getting a computer to understand a doctor and patient is like trying to teach a parrot to recite a poem in a language it's never heard before, while the parrot is sitting in a noisy, crowded market. If the computer gets the words wrong, the medical note is wrong, and that could be dangerous. So, scientists need to figure out which computers are actually good at this tricky task before we let them into real clinics.
This paper is like a massive, high-stakes taste test for ten different "listening brains" (AI models) to see which one is the best at understanding medical conversations in Latin American Spanish. The researchers gathered ten real-life videos of doctors talking to patients from different countries in the region. They hired a human to write down exactly what was said in every video, creating a "gold standard" answer key. Then, they fed the audio of these ten videos into ten different AI models—five that are free and open for anyone to use, and five that are paid, closed-source tools from big tech companies. They scored each AI based on how close their transcription was to the human's perfect version, looking at everything from simple word mistakes to whether the meaning was preserved.
But the researchers didn't just stop at testing; they wanted to see if they could make one of the best open-source models even better by "teaching" it specifically on medical data. They took the top open-source contender, called Whisper Large v3, and gave it a crash course using nine of the ten videos. They chopped the audio into tiny 10-second chunks and let the model practice over and over, trying different teaching methods to see if it could learn the specific slang and medical terms of Latin American doctors. They even tried "data augmentation," which is like a teacher adding background noise, changing the pitch of the voice, or playing two conversations at once to make the student work harder and learn to ignore distractions.
Here is where the story gets interesting. The researchers found that the paid, closed-source models were generally the strongest listeners, with one model called Gemini-2.5-pro coming out on top overall. However, among the free, open-source models, Whisper Large v3 was the clear winner. When they tried to fine-tune this model to make it even better, they hit a surprising snag. They thought that adding all that extra "noise" and practice (data augmentation) would make the model smarter, like a student who learns to study in a chaotic library. Instead, it made the model worse. The extra noise confused the model, and it actually performed better when they just let it study the clean, clear recordings without the added chaos.
In the final showdown, they tested their "trained" model on the one video it had never seen before (the tenth video). Even with its special training, the fine-tuned model didn't beat the top paid models. In fact, on that single unseen video, it ranked fourth out of six when compared to the big commercial tools. However, it did show some promise in specific areas like understanding the overall meaning of sentences, scoring higher than some other models in "semantic similarity." The paper suggests that while fine-tuning Whisper Large v3 is a solid strategy for creating a free medical scribe for Latin America, it's not a magic bullet that instantly beats the expensive, corporate giants. The researchers conclude that while the fine-tuned open-source model is a strong baseline, we still need much more data and better training to truly master the diverse and complex world of Latin American medical speech.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.