Technical Summary: Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR

Problem Statement

Automatic Speech Recognition (ASR) for Swiss German (Schweizerdeutsch) faces a fundamental challenge: the language is a continuum of dialects with no standardized orthography, exhibiting significant phonological, morphological, and syntactic divergence from Standard German (Hochdeutsch). Existing ASR systems, including OpenAI's Whisper, struggle with this dialect due to underrepresentation in training corpora, often producing Standard German approximations, English interpolations, or hallucinated outputs.

A critical issue in the field is the evaluation of these systems. Published state-of-the-art results (ranging from 12.1% to 17.5% Word Error Rate) rely on proprietary data, lack model weight availability, or suffer from benchmark contamination, where evaluation data is inadvertently included in training or shares identical transcription conventions. Furthermore, standard Word Error Rate (WER) is ill-suited for dialect-to-standard translation tasks, as it penalizes valid stylistic variations (e.g., tense shifts, word order changes) as errors, conflating translation ambiguity with recognition failure.

Methodology

Data Collection and Pipeline

The study utilizes 1,367 hours of broadcast speech paired with professionally authored Standard German subtitles as weak supervision. Data is sourced from four publicly accessible repositories:

1. SRF Mediathek: 675.6 hours of diverse Swiss public TV content.
2. Swiss Parliament (SPC v2): 202.4 hours of Bernese dialect debates.
3. YouTube: 132.8 hours from institutional and public channels.
4. PlaySuisse: 179.9 hours comprising scripted series and high-dialect-density films.

A rigorous pipeline handles data processing:

• Quality Filtering: Audio-text pairs are filtered based on silence duration, signal-to-noise ratio (SNR), length, and the presence of meta-text (credits, sound descriptions).
• Subtitle Alignment: A pipeline aligns WebVTT subtitles with audio, though experiments showed that realigning timestamps provided negligible gains over original timestamps, suggesting Whisper's 30-second attention window compensates for minor offsets.
• Curriculum Design: Training runs employed sequential and mixed strategies, isolating the impact of specific data sources (e.g., distinguishing between noisy scripted series and clean dialect films).

Model Architecture and Training

The base model is Whisper large-v3 (1.55B parameters). The study compares two fine-tuning strategies across 16 iterative runs on an NVIDIA DGX Spark (128 GB unified memory):

1. LoRA (Low-Rank Adaptation): Applied to all attention and feed-forward projections. The study systematically varied rank ($r$) and the scaling factor ($\alpha$). A critical finding was that the common LLM heuristic $\alpha = 2r$ causes catastrophic decoder instability (hallucinations) in Whisper. The optimal configuration used a conservative scaling of $\alpha/r = 0.2$.
2. Full Fine-Tuning: All 1.55B parameters were trained using a three-phase curriculum (Foundation, Diversification, Specialization) followed by a mixed epoch.

Evaluation Strategy

Models were evaluated on the All Swiss German Dialects Test Set (ASGDTS), a public benchmark strictly disjoint from training data.

• Metrics: Standard WER was computed, but the authors introduced a Harmonized WER Analysis to decompose errors:
  • cWER (Content WER): Counts only genuine transcription errors (partial or total failures).
  • sWER (Style WER): Counts semantically correct outputs penalized for stylistic differences (tense, word order, Swiss orthography).
  • bWER (Bias-Corrected WER): Adjusts for systematic over-classification of style as error.
• Self-Training Control: To test for benchmark contamination, a "vanilla" Whisper model was fine-tuned exclusively on the ASGDTS test set (zero Swiss German training data) to measure the baseline "memorization" effect.

Key Results

1. Honest Baseline Performance

The best honest models (trained and evaluated on strictly disjoint data) achieved:

• LoRA Adapter: 25.32% WER (13.9% cWER).
• Full Fine-Tuned Model: 25.60% WER (13.8% cWER).
• Improvement: Full fine-tuning provided a modest improvement over the zero-shot baseline (28.56% $\to$ 25.60%), outperforming LoRA in stability and robustness, though both plateaued around 25.6% WER.

2. Decomposition of Error Rates

The study reveals that standard WER significantly overestimates error in dialect-to-standard tasks:

• 11.3–11.9% of the measured WER penalty stems from valid stylistic variations (sWER), not recognition failures.
• The cWER (genuine errors) is 13.8%.
• The bWER (bias-corrected) is 8.5%, suggesting the true error rate is approximately one-third of the measured WER.

3. Benchmark Contamination and Memorization

A controlled experiment demonstrated that benchmark contamination drives published "state-of-the-art" results:

• A vanilla Whisper model self-trained only on the ASGDTS test set (with no Swiss German training data) achieved 13.88% WER.
• This trivially surpasses published results (e.g., 17.1% and 17.5%) and approaches the "honest" cWER of the full fine-tuned model.
• Conclusion: Published low WERs primarily reflect convention matching (memorizing the specific transcription style of the benchmark) rather than genuine dialect comprehension. Prior training on Swiss German data actually hurt self-training performance, indicating that robust models resist overfitting to benchmark conventions.

4. LoRA Scaling and Hallucination

The study identified that excessive LoRA scaling ($\alpha/r = 2.0$) destabilizes the Whisper decoder, causing it to forget end-of-sequence behavior and hallucinate. Reducing scaling to $\alpha/r = 0.2$ eliminated hallucinations and improved performance, correcting a common heuristic misapplication in ASR.

5. Data Quality vs. Quantity

Data quality proved more critical than volume.

• Subtitle Provider Matters: Data from specific providers (e.g., PlaySuisse scripted series with embedded credits) caused performance degradation (+2.92 pp WER), while high-dialect-density films with clean subtitles from the same platform improved performance (-2.28 pp WER).
• Curriculum Limits: Sequential training on diverse data sources often led to catastrophic forgetting, whereas mixed training on high-quality subsets (785h) matched the performance of the full 1,367h dataset.

Significance and Claims

The paper claims to provide the first openly licensed, rigorously evaluated, and reproducible baseline for Swiss German ASR using Whisper. Its primary contributions are:

1. Exposing Benchmark Contamination: It demonstrates that current state-of-the-art results are inflated by training on or matching the conventions of the evaluation set. A model with zero dialect training can achieve "SOTA" WERs simply by memorizing the test set's transcription style.
2. Redefining Evaluation: It argues that raw WER is inadequate for dialect-to-standard translation. The proposed cWER and bWER metrics offer a fairer assessment by separating recognition failures from valid stylistic translation choices.
3. Technical Correction: It corrects the application of LoRA scaling heuristics for encoder-decoder ASR models, preventing decoder instability.
4. Reproducibility: Unlike prior works relying on proprietary data or restrictive licenses, this study releases model weights (Apache 2.0) and uses only publicly accessible data, enabling independent verification.
5. Hardware Accessibility: It demonstrates that billion-parameter ASR fine-tuning is feasible on a single desktop workstation (NVIDIA DGX Spark), democratizing access to high-resource model research.

The authors conclude that the "style floor" (the minimum WER achievable due to translation ambiguity) is intrinsic to the task and cannot be reduced by hyperparameter tuning alone. Future progress requires semantic evaluation metrics and convention-aware training strategies rather than simply chasing lower raw WER numbers.