💬 NLP

When Multiple Scripts Matter: Evaluating ASR in Clinical Settings

This paper introduces MultiClin, a clinical ASR benchmark that addresses the underestimation of performance in non-English settings caused by multiscript variability, demonstrating that multiscript-aware evaluation yields fairer assessments and that unifying scripts during training significantly improves model convergence and recognition accuracy.

Original authors: Jean Seo, Minkyu Kim, Jeonguk Lee, Jisoo Jung, Wooseok Han, Eunho Yang

Published 2026-06-17✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jean Seo, Minkyu Kim, Jeonguk Lee, Jisoo Jung, Wooseok Han, Eunho Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to listen to a doctor talk to a patient. In an English-speaking hospital, this is relatively straightforward: the doctor says "brace," and the robot writes down "brace."

But in many non-English countries, like South Korea, the situation is a bit like a bilingual party where everyone is speaking the same language, just with different accents or spellings.

Here is the story of the paper, broken down into simple concepts:

1. The Problem: The "One Right Answer" Trap

In the real world, a Korean doctor might say the English word "brace" (for a knee support). But when writing it down, they might write it in two valid ways:

Way A: The English spelling: "brace"
Way B: The Korean phonetic sound: "bureseu" (브레이스)

Both mean the exact same thing and sound the same. However, standard computer tests for speech recognition are like a strict teacher who only accepts one specific answer. If the robot writes "bureseu" but the test key says "brace," the computer marks it as wrong, even though the robot understood the doctor perfectly.

The authors call this "Multiscript Variability." It's like if you asked a friend to write down "Hello," and they wrote "Hullo" or "Salut" (if they were French), and you told them they failed the test just because the spelling wasn't exactly what you had in mind.

2. The Solution: The "MultiClin" Benchmark

The researchers built a new testing ground called MultiClin. Think of this as a specialized exam for speech robots that understands the "bilingual party" rule.

The Dataset: They created a library of fake (but realistic) doctor-patient conversations. Because real medical recordings are private (like a secret diary), they used AI to generate these conversations, carefully adding in medical terms that could be written in English or Korean.
The New Rule: Instead of checking if the robot's answer matches one specific script, the new test checks if the robot's answer matches either the English version or the Korean version. It's like a teacher saying, "If you wrote 'brace' OR 'bureseu', you get full credit."

3. The Results: Robots Look Much Smarter

When the researchers tested popular speech robots (like Whisper, Qwen, and Gemini) using the old "strict teacher" rules, they looked terrible. The error rates were high because the robots were being punished for using the "correct" local spelling.

But when they used the new MultiClin rules:

The error rates dropped significantly.
The robots weren't actually dumber; the test was just too harsh.
The best robot (Gemini 2.5 Pro) showed it could handle these tricky medical conversations much better than we thought, once we stopped penalizing it for using the local script.

4. The Training Lesson: Pick a Lane!

The researchers also tried to teach the robots themselves using this new data. They discovered a very important lesson about how to train them: Consistency is key.

Imagine you are teaching a child to write the word "cat."

Scenario A: You show them "cat" 100% of the time. They learn it perfectly.
Scenario B: You show them "cat" 50% of the time and "kæt" (phonetic) the other 50%. The child gets confused. They don't know which one is the "real" word, and they start making mistakes.

The paper found that if the training data mixed English and Korean spellings randomly (a 50/50 split), the robot got very confused and performed poorly. It was like the robot's brain was spinning in circles trying to decide which spelling was right.

The Winner: The robots performed best when the training data was 100% unified. If the goal was to write in Korean, everything was written in Korean. If the goal was English, everything was English. This removed the confusion and let the robot learn the medical terms quickly and accurately.

Summary

The Issue: Current tests unfairly punish speech robots for using local spellings of medical words, even when they are correct.
The Fix: The authors created MultiClin, a new test that accepts multiple valid spellings (English or local script) as correct answers.
The Discovery: Robots are actually much better at understanding medical speech than we thought, but we need to stop grading them with a "one-size-fits-all" ruler.
The Training Tip: To teach these robots well, don't mix spelling styles randomly. Pick one style and stick to it, or the robot gets confused.

Technical Summary: Evaluating ASR in Clinical Settings with Multiscript Variability

Problem Statement
Automatic Speech Recognition (ASR) in non-English clinical environments faces a specific challenge known as multiscript variability. Unlike standard code-switching, which involves acoustic alternation between languages, multiscript variability occurs when a single spoken medical term corresponds to multiple valid orthographic forms (e.g., an English medical term written in the Roman alphabet versus its phonetic rendering in a local script like Korean Hangul).

Conventional ASR evaluation metrics, such as Word Error Rate (WER), rely on a single-reference transcription assumption. In clinical settings, this assumption fails because English-origin medical terms often lack standardized localization guidelines, leading to multiple valid transcriptions. Strict string-matching metrics treat these valid orthographic variants as errors, systematically underestimating ASR performance. Furthermore, existing benchmarks and metrics (like Transliterated WER) have primarily focused on general-domain code-switching or dialectal variation, leaving clinical multiscript settings largely unexplored.

Methodology
To address these gaps, the authors introduce MultiClin, a clinical ASR benchmark designed to evaluate robustness to multiscript variability.

Dataset Construction: The dataset was built from publicly available doctor-patient dialogues (ACIBench, Primock57, MTS-Dialog). The process involved:
1. Tagging: Using an LLM to identify and tag script-switching instances into three categories: MEDICAL (English terms), UNIT (measurement units), and NUMBER.
2. Translation & Transliteration: Dialogues were translated into Korean. Crucially, tagged entities were preserved in their original form but augmented with a phonetic rendering in the local script (e.g., "injection, 인젝션"), separated by commas. This creates a "many-to-one" mapping between speech and valid orthography.
3. Human Annotation: Nursing professionals reviewed the data for orthographic correctness and naturalness.
4. Speech Synthesis: To comply with HIPAA and privacy restrictions, audio was synthesized using TTS models with specific speaker styles (professional doctors, lethargic patients) and simulated clinical noise (reverb, HVAC).
5. Statistics: The final dataset contains 316 dialogues covering 20+ clinical specialties, with an average of 34 turns per dialogue.
Evaluation Protocol (Dynamic Multiscript Reference Resolution): The authors propose a localized evaluation metric (Algorithm 1) that moves beyond single-reference matching.
- For each tagged entity in the reference, the system dynamically extracts a 50-character window from the ASR prediction.
- It computes the Character Error Rate (CER) for both the original English form and the local script form against this window.
- The system selects the reference variant (original or transliterated) that yields the minimal local error, effectively treating both orthographic forms as valid ground truth.
Experimental Setup:
- Models: Zero-shot inference was tested on Whisper (v3, v3-turbo), Qwen3 ASR (0.6B, 1.7B), and Gemini (2.5 Flash, 2.5 Pro).
- Fine-tuning: Whisper models were fine-tuned using LoRA. The study specifically investigated the impact of labeling consistency by varying the "transliteration ratio" (the proportion of tagged entities rendered in the local script vs. the original script) during training.

Key Results

Evaluation Bias: Moving from strict single-label matching ("original" only) to multiscript-aware evaluation ("both") significantly reduced error rates across all models. For example, Gemini 2.5 Pro's WER dropped from 28.28% to 15.78% when medical terms were evaluated with multiscript flexibility. This confirms that conventional metrics systematically penalize correct, phonetically accurate outputs that differ orthographically from a single reference.
Model Performance: Under the multiscript-aware setting, Gemini 2.5 Pro achieved the best Character Error Rate (CER) of 4.86%. Among open-source models, Whisper v3 Turbo demonstrated the strongest robustness (23.00% WER).
Fine-tuning Gains: Fine-tuning Whisper models on the MultiClin dataset with a 100% transliteration ratio (unifying all tagged entities into the local script) yielded substantial improvements. Whisper-Large v3 Turbo achieved a best-in-class CER of 6.16% (a 3.83% absolute reduction over its pre-trained baseline).
Impact of Labeling Consistency: The study found a non-monotonic relationship between transliteration ratio and performance.
- 0% Ratio (Mixed/Roman only): Highest error rates (69.17% CER).
- 50% Ratio: A secondary error peak (57.47% CER). The authors attribute this to orthographic uncertainty, where inconsistent script mappings maximize conditional entropy $H(Y|X)$ , disrupting the model's ability to form stable decision boundaries.
- 100% Ratio (Unified): Consistently yielded the best performance (7.66% CER), validating that script unification provides a deterministic learning signal.

Significance and Claims
The paper claims that MultiClin provides a fairer and more informative evaluation framework for non-English clinical ASR by accounting for valid orthographic variations that traditional metrics miss. The authors emphasize that:

Multiscript-aware evaluation reveals the true capabilities of ASR models, which are often underestimated by strict string-matching.
Script unification (training with consistent orthography) is an essential strategy for mitigating orthographic ambiguity and improving model convergence in clinical settings.
The dataset and code are publicly available to facilitate further research into multiscript variability and clinical ASR robustness.

The work concludes by noting that future research should examine how these ASR improvements influence downstream clinical tasks, such as entity extraction and SOAP note generation.