Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis

1. Problem Statement

Taiwanese Hokkien (Taigi) is a linguistically significant language with deep cultural roots in Taiwan, yet it suffers from a lack of robust speech technology compared to English or Mandarin. Key challenges include:

• Data Scarcity: Limited availability of high-quality, labeled speech corpora.
• Linguistic Complexity: Taigi features complex phonological systems, diverse regional accents, intricate tone sandhi patterns, and multiple orthographic conventions.
• Evaluation Gap: There are no standardized, reproducible benchmarks for Taigi Automatic Speech Recognition (ASR) or Text-to-Speech (TTS). Existing systems are often proprietary or adapted from general multilingual models without rigorous evaluation protocols, making objective comparison impossible.
• Code-Switching Nuances: Natural Taigi speech often involves code-switching to Mandarin for technical terms or proper nouns, complicating the definition of "ground truth" and "authenticity."

2. Methodology

The authors introduce Breeze Taigi, a comprehensive framework comprising standardized benchmarks, evaluation methodologies, and baseline models.

A. Benchmark Construction

• Data Source: The benchmark utilizes 30 carefully curated audio pairs from the Executive Yuan's Public Service Announcements (PSA). These include parallel Mandarin and Taigi audio files with official scripts.
• Ground Truth Generation:
  • ASR Benchmark: Instead of relying solely on difficult Taigi transcriptions, the authors map Taigi audio to Mandarin text transcriptions. They leverage the parallel Mandarin audio, refine it using Large Language Models (LLMs) and state-of-the-art Mandarin ASR, and verify it with human experts.
  • Normalization: To ensure fair comparison across systems, output text is normalized (handling number formats, punctuation, and case sensitivity).
• Evaluation Metrics:
  • ASR: Character Error Rate (CER) is used as the primary metric. CER is preferred over Word Error Rate (WER) because Chinese characters lack natural word boundaries, and segmentation algorithms vary.
  • TTS: A dual evaluation framework is employed:
    1. Automatic: ASR-based transcription of synthesized speech compared to input text (CER).
    2. Human Evaluation: Three dimensions:
       • CER: Transcription accuracy.
       • Taiwanese Pronunciation Accuracy: Percentage of characters pronounced with correct Taigi phonetics (including tone sandhi).
       • Mean Opinion Score (MOS): Naturalness rating (1–5 scale).

B. Model Development & Training

• Data Generation: To overcome data scarcity, the authors generated a massive synthetic dataset of ~10,000 hours of Taigi speech. This dataset captures diverse speakers, acoustic environments, and conversational contexts.
• ASR Model (BreezeASR-Taigi):
  • Architecture: Based on Whisper (a large-scale multilingual model).
  • Training: Fine-tuned on the 10,000-hour synthetic Taigi dataset.
  • Tokenization: Uses Whisper's multilingual tokenizer.
• TTS Model (BreezyVoice-Taigi):
  • Architecture: Based on CosyVoice 2, a streaming synthesis framework using Large Language Models (LLMs) and finite scalar quantization.
  • Training: Fine-tuned on the same 10,000-hour synthetic dataset. Only the LLM component was fine-tuned; other modules were frozen.

3. Key Contributions

1. Standardized Benchmarks: Established the first reproducible evaluation protocols for Taigi ASR and TTS, utilizing parallel Mandarin-Taigi resources to create reliable ground truth.
2. Large-Scale Synthetic Data: Created a 10,000-hour diverse synthetic speech dataset, significantly expanding available resources for low-resource language research.
3. Reference Implementations: Released open baseline models (BreezeASR-Taigi and BreezyVoice-Taigi) that outperform existing commercial and research systems.
4. Methodological Insight: Demonstrated that mapping Taigi audio to Mandarin text is a pragmatic and effective strategy for benchmarking ASR, despite linguistic differences.
5. Sociolinguistic Discovery: Highlighted the trade-off between "naturalistic" speech (which includes code-switching to Mandarin) and "prescriptive" phonological purity, showing that high naturalness scores can coexist with lower strict Taigi pronunciation scores.

4. Results

ASR Performance (30 Test Samples)

• BreezeASR-Taigi: Achieved the lowest Average CER of 30.13%.
• Comparison: Outperformed the commercial system "Yating" (32.11%), the Ministry of Education's Input Method (30.70%), and the Mandarin-optimized "ASR25" (49.99%).
• Insight: The model showed consistent performance, whereas Mandarin-optimized models struggled significantly with Taigi-specific phonetic patterns.

TTS Performance (20 Test Samples)

• Automatic Evaluation (CER):
  • BreezyVoice-Taigi: 19.09% (Best).
  • Aten AI Voice: 23.14%.
  • Taigi AI Labs: 38.19%.
• Human Evaluation:
  • Naturalness (MOS): BreezyVoice-Taigi achieved a perfect 5.0, indicating speech indistinguishable from humans.
  • Pronunciation Authenticity: BreezyVoice-Taigi scored 59.2% on strict Taigi pronunciation.
  • Analysis: BreezyVoice-Taigi frequently code-switched to Mandarin for proper nouns/technical terms (reflecting natural bilingual usage), which lowered its "pure Taigi" score but increased naturalness and intelligibility. In contrast, Aten AI Voice scored highest on pronunciation purity (89.8%) but lower on naturalness (3.8 MOS).

5. Significance

• Advancing Low-Resource Languages: The paper provides a replicable framework for developing speech technologies for languages with limited data by leveraging parallel resources and large-scale synthetic data generation.
• Redefining Evaluation: It challenges the notion that "perfect" TTS must adhere strictly to prescriptive phonology. The results suggest that for conversational AI, naturalness and intelligibility (which may include code-switching) are often more valuable than rigid phonological purity.
• Community Resource: By open-sourcing benchmarks, datasets, and models, the authors enable the research community to objectively measure progress and innovate in Taigi speech technology, potentially setting a precedent for other under-resourced languages.
• Pragmatic Benchmarking: The strategy of using parallel Mandarin resources to evaluate Taigi systems offers a cost-effective solution for creating high-quality benchmarks where native transcriptions are scarce.