ViMedCSS: A Vietnamese Medical Code-Switching Speech Dataset & Benchmark

1. Problem Statement

Code-switching (CS) is a prevalent phenomenon in Vietnamese medical communication, where healthcare professionals frequently insert English terms (e.g., drug names, anatomical structures, procedures) into Vietnamese sentences.

• The Challenge: Current Automatic Speech Recognition (ASR) systems struggle to accurately transcribe these embedded English terms within Vietnamese contexts.
• The Gap: There is a lack of open benchmarks specifically designed for Vietnamese medical code-switching. Existing systems face a trade-off: models optimized for Vietnamese minimize errors in the "matrix" language (Vietnamese) but fail on English insertions, while multilingual models recognize English better but often degrade overall Vietnamese accuracy.
• Consequence: Misrecognition of critical medical terms leads to errors in clinical documentation, medication administration, and medical education.

2. Methodology

A. Dataset Construction (ViMedCSS)

The authors constructed ViMedCSS, the first public benchmark for this domain, through a rigorous pipeline:

• Source: A curated bilingual medical lexicon (Meddict) containing 64,232 entries. They selected 3,203 terms that retain English/foreign roots in Vietnamese usage.
• Data Collection:
  • Searched public platforms (YouTube) using the CS terms as queries.
  • Filtered candidates using Large Language Models (LLMs) to ensure Vietnamese titles and medical domain relevance.
  • Processed over 700 hours of raw audio.
• Transcription & Alignment:
  • Used Gemini 2.5 Pro to transcribe audio, segment it into 5–15 second clips, and automatically identify CS spans and non-Vietnamese terms.
  • Applied semantic filtering to remove non-medical or non-Vietnamese utterances.
  • Normalized surface forms and aligned detected terms to the dictionary using Levenshtein distance to handle orthographic variations.
• Final Corpus:
  • 34.6 hours of audio.
  • 16,576 utterances.
  • 5 Medical Topics: Medical Sciences, Pathology & Pathogens, Treatments, Nutrition, and Diagnostics.
  • Guarantee: Every utterance contains at least one code-switched medical term.
• Splits: The data is divided into Train (80%), Validation (10%), Test (10%), and a dedicated Hard Split containing only rare/unseen terms (occurring 1–2 times in the whole dataset) to test generalization.

B. Evaluation Metrics

To disentangle performance on the matrix language vs. the code-switched spans, the authors use:

• WER/CER: Standard Word/Character Error Rates on the full transcript.
• CS-WER: Error rate calculated only on code-switched tokens.
• N-WER: Error rate calculated on non-code-switched tokens.

C. Experimental Setup

• Baselines: Evaluated zero-shot performance of state-of-the-art models:
  • Multilingual: Whisper (Small/Large-v3), MMS.
  • Vietnamese-optimized: PhoWhisper, wav2vec2-base-vi, VietASR.
• Fine-tuning Strategies: Using PhoWhisper-small as the backbone, the authors tested four adaptation families:
  1. Contextual Biasing (Decoder-side): Dynamic Vocabulary (DV) and Rank & Selection (RS).
  2. Post-processing: AdaCS (bias-attention normalization).
  3. Parameter-Efficient Adapters: LoRA (Low-Rank Adaptation).
  4. Language Identity (LID) Guidance: Attention Guide (AG) to steer language-specific attention heads.
  • Hybrids: Combinations like LoRA+AdaCS and AG+AdaCS.

3. Key Results

Zero-Shot Performance

• Trade-off Confirmed: Vietnamese-optimized models (e.g., VietASR, PhoWhisper) achieved superior overall WER and N-WER (Vietnamese text accuracy) but higher CS-WER (English term errors).
• Multilingual Strength: Large multilingual models (Whisper-Large-v3) achieved the lowest CS-WER but slightly higher overall error rates compared to specialized Vietnamese models.

Fine-Tuning Performance

• Contextual Biasing Alone: Methods like Dynamic Vocabulary and Rank & Selection offered only marginal improvements. AdaCS significantly reduced CS-WER but degraded overall WER (precision-recall trade-off).
• Adapter-Based Methods: LoRA and Attention Guide (AG) yielded the most significant improvements across all metrics.
  • AG achieved the best overall results: Test WER of 23.67% and CS-WER of 19.50%.
  • LoRA was a strong second, reducing CS-WER by roughly half compared to the frozen model.
• Hybrid Approaches: Combining adapters with post-processing (e.g., AG+AdaCS) stabilized performance, recovering sentence-level accuracy while maintaining low CS-WER.
• Initialization Matters: Fine-tuning a Vietnamese-optimized backbone (PhoWhisper) consistently outperformed fine-tuning a multilingual backbone (Whisper) on both general and code-switched metrics.

Topic Analysis

• Treatments was the most error-prone category initially but improved significantly after fine-tuning.
• Medical Sciences remained the most challenging due to its diversity and volume.
• AG effectively narrowed the performance gap across all five medical topics.

4. Key Contributions

1. ViMedCSS Dataset: The first large-scale, publicly available benchmark (34.6 hours, 16k+ utterances) specifically for Vietnamese medical code-switching, featuring a "Hard Split" for rare term evaluation.
2. Systematic Benchmarking: Established zero-shot baselines and a comprehensive evaluation framework separating matrix language accuracy from code-switched span accuracy.
3. Methodological Insights: Demonstrated that parameter-efficient adaptation (specifically Attention Guide) on a domain-specialized backbone is the most effective strategy for low-resource, code-switching ASR, outperforming pure decoder-side biasing or multilingual pretraining alone.

5. Significance

• Clinical Impact: Provides a pathway to more accurate automated transcription for medical records, reducing risks of medication errors and improving the quality of digital health data in Vietnam.
• Technical Advancement: Offers a clear solution to the "multilingual vs. monolingual" trade-off in ASR, showing that combining domain-specific pretraining with LID-guided fine-tuning yields the best of both worlds.
• Community Resource: The dataset and benchmarks enable future research into contextual biasing, domain adaptation, and low-resource multilingual speech processing.