LuxEmo: Expressive Text-to-Speech Corpus for Luxembourgish

Technical Summary: LuxEmo – Expressive Text-to-Speech Corpus for Luxembourgish

Problem Statement
State-of-the-art speech datasets and expressive Text-to-Speech (TTS) systems predominantly focus on high-resource languages (e.g., English) and controlled studio recordings. Low-resource languages, such as Luxembourgish, remain underrepresented. Luxembourgish presents specific challenges: it is embedded in a multilingual ecology with frequent code-switching (German, French, English), and existing resources lack spontaneous, conversational data with emotional prosody. Current multilingual TTS approaches often rely on high-resource surrogate languages or transfer-learning strategies that fail to capture the specific linguistic and prosodic nuances of Luxembourgish, particularly in realistic media conditions involving noise, overlapping speech, and informal registers.

Methodology
The paper introduces LuxEmo, a 21-hour spontaneous emotional speech corpus derived from Radio Télévision Luxembourg (RTL) youth broadcasts. The work focuses on resource and protocol construction rather than novel architectural design.

1. Corpus Construction & Curation:

   • Source: RTL Youth programs featuring studio discussions, interviews, and on-location reports.
   • Preprocessing: Audio was extracted, resampled to wideband, and processed using Voice Activity Detection (VAD) to remove non-speech segments. DeepFilterNet was applied for denoising (attenuating background music and ambient noise).
   • Segmentation: Segments were limited to 1–10 seconds (mean duration 9.87s). Transcriptions were obtained via LuxASR (a Whisper large-v3 model fine-tuned for Luxembourgish) and aligned with VAD segments.
   • Speaker Identification: Diarization output was mapped to four recurring speakers (1 male, 3 female).
   • Annotation:
     • Language Identification (LID): A Wav2Vec2-based model (mms-lid-4017) classified utterances. 83.8% were Luxembourgish, with 6.5% English and 2.2% German. A native speaker manually reviewed ~15% of segments to validate assignments.
     • Emotion Detection: A HuBERT-based classifier and lexical models flagged four categories: neutral, happy, sad, and angry. Labels are treated as "weakly supervised" with confidence scores; high-confidence items form the supervised subset, while lower-confidence segments support representation learning.
   • Data Splits: A speaker-aware and episode-aware splitting strategy was employed to prevent data leakage from shared conversational contexts. Splits maintain global emotion distribution and ensure coverage of all four emotion classes in the test set.

2. TTS Benchmarking:
   Five expressive TTS systems were evaluated to cover different deployment strategies:

   • GradTTS: Diffusion-based model using German as a proxy language.
   • XTTS: Cross-lingual reference-based model using German as a proxy.
   • Toucan: Reference encoder-based model with a multilingual phoneme inventory covering Luxembourgish.
   • Qwen3 FT: A prompt-controlled, language-model-based TTS adapted (fine-tuned) to LuxEmo.
   • kNN TTS: A non-parametric baseline transferring prosody via k-nearest neighbor retrieval from LuxEmo.

3. Evaluation Protocol:

   • Objective Metrics: Included non-intrusive quality prediction (WV-MOS, NISQA-TTS, NISQA v2.0, DNSMOS), intelligibility (WER/CER via Whisper and LuxASR), speaker similarity (ECAPA-TDNN embeddings), and prosody (F0 RMSE, Pearson correlation).
   • Subjective Evaluation: A listening test with 20 native Luxembourgish listeners rated naturalness, emotion appropriateness, and intensity on 5-point scales.

Key Contributions

• LuxEmo Dataset: The first large-scale (21 hours, 7,562 segments), spontaneous, emotional speech corpus for Luxembourgish, annotated for language, speaker, and four emotion categories in a code-switching environment.
• Curation Workflow: A documented semi-automatic pipeline combining VAD, denoising, LuxASR-based segmentation, acoustic/lexical emotion cues, and targeted human review, suitable for broadcast archives with practical reuse constraints.
• Benchmarking: A comprehensive evaluation of five TTS strategies (zero-shot cross-lingual, multilingual, and adapted) under realistic, heterogeneous conditions.
• Standardized Splits: Speaker- and episode-aware train/dev/test splits designed to minimize leakage while preserving emotion coverage.

Results

• Quality vs. Intelligibility Trade-offs: No single model dominated across all metrics.
  • GradTTS (German proxy) achieved the highest waveform smoothness (WV-MOS) but the highest Luxembourgish WER, indicating that smooth waveforms do not guarantee target-language segmental accuracy.
  • Toucan (Luxembourgish-aware) achieved the lowest LuxASR WER and high overall quality, demonstrating the value of language-specific phoneme coverage.
  • XTTS (German proxy) showed high speaker similarity but intermediate intelligibility.
• Adaptation Effects:
  • Qwen3 FT (Fine-tuned) received the highest subjective ratings for naturalness and emotion appropriateness from native listeners, despite having lower objective quality scores (NISQA/DNSMOS) and higher ASR-based WER than zero-shot systems. This suggests perceived expressiveness does not always align with ASR-derived transcription accuracy.
  • kNN TTS successfully retrieved prosody (comparable F0 RMSE to GradTTS) but was limited by poor acoustic waveform quality.
• Emotion Performance: "Angry" and "Sad" emotions were consistently harder to synthesize, correlating with the corpus imbalance (anger represents only 0.5% of segments).
• Metric Divergence: Objective metrics (WV-MOS, NISQA, DNSMOS) emphasized different aspects of quality (e.g., waveform smoothness vs. noise suppression), reinforcing the need for multi-metric evaluation.

Significance and Claims
The paper positions LuxEmo as a testbed for expressive TTS in low-resource, code-switching settings, rather than a definitive solution for universal TTS performance.

• Realistic Constraints: The work highlights that clean recordings, balanced labels, and broad speaker coverage are unavailable in realistic low-resource settings; LuxEmo addresses this by documenting curation strategies for noisy, spontaneous broadcast data.
• Limitations of Surrogates: The results indicate that German-proxy models, while improving perceived quality, fail to fully capture Luxembourgish-specific phonotactics and expressive cues.
• Modest Scope: The authors explicitly state that the benchmark characterizes realistic system behavior under heterogeneous conditions and should not be interpreted as evidence for a universally best TTS model. The four-speaker composition is a result of source material availability, not demographic sampling, meaning the corpus is not demographically representative of all Luxembourgish speakers.
• Future Direction: The paper concludes that future work should broaden speaker and domain coverage, refine emotion labels through inter-annotator analysis, and adapt the pipeline to other low-resource, code-switching environments.