Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition

Technical Summary: Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition

Problem Statement
Automatic Speech Recognition (ASR) systems struggle with code-switching (CS), where speakers alternate between languages within a single utterance. While standard fine-tuning reduces overall error rates, recognition errors remain disproportionately concentrated in CS-critical regions, specifically embedded-language words and switch-boundary neighborhoods. Existing approaches often lack explicit signals to target these confusable spans, and standard objectives fail to distinguish between the ground-truth transcription and acoustically plausible but incorrect alternatives ("near-misses").

Methodology
The authors propose a Point-of-Interest (POI)-aware contrastive training framework centered on a pipeline called CS-NMG (Code-Switching Near-Miss Generation). The methodology consists of three primary stages:

1. POI Identification and Near-Miss Generation:

   • POI Detection: The system identifies CS spans (embedded-language segments) and their immediate switch-boundary neighborhoods in the reference transcripts.
   • Candidate Construction: Near-miss hypotheses are constructed by perturbing only the identified POIs. This involves:
     • Extracting N-best hypotheses from a fixed seed ASR model.
     • Using an external Large Language Model (LLM) to generate additional POI replacement candidates, ensuring sentence structure is preserved while introducing phonetic confusions localized to the POI.
   • Tri-Level Filtering: To ensure the generated near-misses are "hard but plausible," a filtering gate is applied using three constraints:
     • Acoustic: The near-miss must have a log-probability score close to the best N-best hypothesis.
     • Phonetic: The replacement must be phonetically similar to the original (measured via Levenshtein distance on phoneme sequences).
     • Textual: The replacement must be sufficiently different in surface form to constitute a challenging negative.

2. Contrastive Alignment Training:

   • The ASR model (Whisper-small fine-tuned with LoRA) is optimized using a dual-objective loss function:
     • Anchor Objective: A POI-weighted Cross-Entropy (WCE) loss that upweights tokens within POI spans.
     • Contrastive Ranking Loss: An InfoNCE-style loss that explicitly ranks the ground-truth transcript higher than the selected hard negative near-misses. This encourages the model to prefer the correct CS terms over acoustically plausible but incorrect transcriptions.

3. Evaluation Metrics:

   • Performance is measured using standard Word Error Rate (WER) and Point-of-Interest Error Rate (PIER), which isolates errors occurring specifically within the identified CS spans.

Key Contributions

• CS-NMG Pipeline: An acoustic-aware mechanism for generating POI-local near-misses seeded from N-best lists and expanded by LLMs, preserving audio plausibility while introducing targeted confusions.
• Contrastive Training Objective: A novel training strategy that combines POI-weighted cross-entropy with multi-negative contrastive ranking. Unlike simple token reweighting, this approach explicitly separates common POI confusions by learning to rank the reference above plausible alternatives.
• Empirical Validation: The framework demonstrates consistent improvements over strong baselines (including standard fine-tuning, WCE, and sequence-level MWER) on both general and domain-specific code-switching benchmarks.

Experimental Results
The method was evaluated on two benchmarks: CS-FLEURS (Mandarin-English) and ViMedCSS (Vietnamese-English, medical domain).

• Performance Gains: The full model (WCE + Contrastive Loss with tri-level filtering) achieved the lowest error rates on both datasets.
  • On CS-FLEURS, it reduced WER to 14.06% and PIER to 15.10%, outperforming the best baseline (MWER) which had a WER of 15.75% and PIER of 16.41%.
  • On ViMedCSS, it reduced WER to 21.87% and PIER to 18.74%, compared to the baseline MWER WER of 23.82% and PIER of 20.84%.
• Ablation Insights:
  • Simply upweighting POI tokens (WCE) yielded only marginal improvements.
  • LLM expansion alone (without filtering) increased the number of negatives but did not consistently improve performance, sometimes degrading WER or PIER.
  • The tri-level filter (Acoustic + Phonetic + Text) was critical; it selected a smaller but higher-quality set of negatives (~3.8 per utterance), leading to the best overall results. This confirmed that effective contrastive learning requires negatives that are acoustically plausible, phonetically similar, yet textually distinct.

Significance and Claims
The paper claims that the primary source of CS errors lies in the inability of models to disambiguate between the correct embedded terms and acoustically similar alternatives. By shifting from a standard likelihood objective to a contrastive ranking objective focused on POI-local near-misses, the proposed framework provides a more informative training signal.

The authors emphasize that their approach improves robustness in CS regions without requiring auxiliary modules during inference, maintaining a standard ASR-only decoding process. They acknowledge limitations regarding the reliance on external LLM APIs for offline data generation and the current scope limited to two language pairs and a single backbone, suggesting future work in open-source candidate expansion and broader language coverage.