TaigiSpeech: A Low-Resource Real-World Speech Intent Dataset and Preliminary Results with Scalable Data Mining In-the-Wild

1. Problem Statement

The paper addresses the critical gap in Spoken Language Understanding (SLU) for low-resource, unwritten languages, specifically Taiwanese Hokkien (Taigi).

• Demographic Urgency: While Mandarin is the dominant language in Taiwan, 64.9% of residents aged 65+ primarily speak Taiwanese Hokkien. As the population ages, there is an increasing need for voice-enabled smart home and healthcare systems (e.g., emergency detection, fall alerts) that cater to this demographic.
• Resource Scarcity: Existing SLU datasets focus on high-resource languages (English, Mandarin, French). Taiwanese Hokkien lacks:
  • Standardized orthography (making text-based mining difficult).
  • Reliable Automatic Speech Recognition (ASR) systems.
  • Real-world, scenario-driven datasets collected from elderly speakers (most existing data is read speech or scripted TV dialogue).
• Data Scarcity Challenge: Collecting labeled data for low-resource languages is expensive and slow. The paper investigates whether scalable data mining from "in-the-wild" sources (e.g., online videos) can serve as a viable alternative to manual collection.

2. Methodology

A. The TaigiSpeech Dataset (Ground Truth)

The authors constructed a new, real-world dataset specifically for elderly speakers in healthcare and home-assistant scenarios.

• Participants: 21 elderly speakers (ages 54–78, mean age 67.9; 13 female, 8 male).
• Content: 3,079 utterances (approx. 6.1 hours) covering 8 intent categories:
  • Emergency (4): SOS Call, Fall Help, Breathing Emergency, General Pain.
  • Non-Emergency/Functional (4): Call Contact, Light On, Light Off, Cancel Alert.
• Collection Protocol:
  • Participants used a custom web-based app on various devices.
  • Scenario-Driven: Instead of reading scripts, participants responded to imagined scenarios (e.g., "Imagine you fell down the stairs...") generated by an LLM.
  • Multimodal Cues: Silent video stimuli (10s) were provided to help visualize the context, though speech was spontaneous.
  • Quality Control: Ambient noise levels were measured pre-recording; data was collected in quiet rooms under researcher supervision.

B. Data Mining Strategies (Scalable Alternatives)

To address the lack of labeled data, the authors explored two strategies to mine training data from a large pool of Taiwanese Drama videos (approx. 7,000 hours) which contain Mandarin subtitles.

1. Keyword Match Mining (Strong Supervision via Pivot Language):

   • Mechanism: Uses Mandarin subtitles as a pivot.
   • Process:
     1. Construct keyword sets for each intent in Mandarin (e.g., "help," "ambulance").
     2. Retrieve subtitle sentences containing these keywords and their context windows.
     3. Use an LLM (Gemini) to perform pseudo-labeling on the context to verify if it matches the target intent.
     4. Extract the aligned audio segment.
   • Output: ~11.3k training samples for a 5-class task (4 emergency + 1 non-emergency).

2. Audio-Visual Mining (Weak Supervision):

   • Mechanism: Relies on multimodal cues without requiring paired text in the target language.
   • Process:
     1. Uses a pre-trained cross-modal encoder (PE-AV) to map video clips (audio + visual) and text intent descriptions into a shared embedding space.
     2. Retrieves video clips that are semantically similar to intent descriptions (written in English/Mandarin).
     3. Aggregates similarity scores to retrieve coarse-grained emergency clips.
   • Output: ~3.6k training samples for a binary task (Emergency vs. Non-Emergency).

C. Experimental Setup

• Models: Evaluated lightweight models (MatchboxNet) and Self-Supervised Learning (SSL) models (HuBERT, WavLM).
• Baselines: Compared against a Cascade ASR + LLM approach (transcribing Taigi to text, then using an LLM for intent).
• Evaluation:
  • Drama Set: Evaluated on the mined data itself (pseudo-labeled).
  • Taigi Set: Evaluated on the real-world TaigiSpeech test set (held-out speakers).
  • Taigi-Adapt: Fine-tuned models on a small subset of TaigiSpeech (10 speakers) to test domain adaptation.

3. Key Results

A. Significant Domain Mismatch

• Performance Drop: Models trained on mined "in-the-wild" data (Drama) performed significantly worse when tested on real-world elderly recordings (Taigi).
  • Example: WavLM-large dropped from 92.36% (Drama) to 70.00% (Taigi) in the Keyword Mining scenario.
  • Example: Audio-Visual mining models dropped to near-random performance (~50%) on the Taigi binary task.
• Implication: The acoustic characteristics, prosody, and vocabulary of scripted drama differ drastically from spontaneous, distressed speech by the elderly. Mined data alone is insufficient for robust deployment.

B. Effectiveness of Domain Adaptation

• When models pre-trained on mined data were fine-tuned (Taigi-adapt) with a small amount of real TaigiSpeech data (1,600 utterances), performance recovered substantially.
  • Result: HuBERT-base and WavLM-base-plus achieved ~90% accuracy on the 8-class Taigi test set.
• Conclusion: Mined data is useful for pre-training, but real-world labeled data is essential for final adaptation.

C. Model Efficiency vs. Performance

• SSL Models vs. Lightweight Models: Large SSL models (HuBERT/WavLM) vastly outperformed lightweight MatchboxNet models, highlighting the need for powerful representations in low-resource settings.
• End-to-End vs. Cascade:
  • End-to-End (SSL): Achieved ~90% accuracy with ~94M parameters.
  • Cascade (ASR + LLM): Even with strong foundation models (Whisper/Qwen3-ASR), the cascade approach achieved only 63–74% accuracy and required significantly more parameters (billions). This suggests end-to-end SLU is more efficient and effective for this specific low-resource task.

4. Key Contributions

1. TaigiSpeech Dataset: The first real-world, scenario-driven spoken intent dataset for Taiwanese Hokkien, specifically targeting the elderly demographic with 3k+ utterances across 8 critical intents.
2. Scalable Data Mining Framework: Demonstrated two strategies (Keyword Match with LLM pseudo-labeling and Audio-Visual mining) to leverage large-scale video data for low-resource languages, while highlighting their limitations.
3. Empirical Benchmark: Provided a realistic evaluation showing that "in-the-wild" mined data suffers from severe domain mismatch with elderly speech, establishing TaigiSpeech as a necessary benchmark for future research.
4. Open Release: The dataset will be released under CC BY 4.0 to facilitate research on low-resource and unwritten languages.

5. Significance

• Social Impact: Directly addresses the "digital divide" for the elderly in Taiwan, enabling voice-activated emergency and home assistance systems for a population that often cannot use Mandarin or touch interfaces effectively.
• Technical Insight: The paper challenges the assumption that large-scale mined data can fully replace manual collection for low-resource SLU. It proves that while mining helps with pre-training, domain-specific, real-world data collection remains irreplaceable for high-stakes applications like healthcare.
• Future Direction: Sets a precedent for building inclusive speech technologies for other unwritten or low-resource languages by combining targeted data collection with scalable mining strategies.