💬 NLP

LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering

This paper demonstrates that parameter-efficient Spoken Question Answering for the low-resource language Luxembourgish can be effectively achieved by training on synthetic speech generated from translated text QA pairs using multiple TTS systems, revealing that task-specific performance depends more on diverse voice configurations than on standard TTS quality metrics.

Original authors: Nina Hosseini-Kivanani, Marco Matassoni, Alessio Brutti

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nina Hosseini-Kivanani, Marco Matassoni, Alessio Brutti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to build a smart assistant that can answer questions spoken in Luxembourgish, a language spoken by only a few hundred thousand people. The problem? To teach a computer to understand spoken language, you usually need thousands of hours of real people recording questions and answers. But for Luxembourgish, that data simply doesn't exist yet.

This paper, LuxSQA, asks a clever question: Can we cheat by using "fake" voices (Text-to-Speech or TTS) to teach the computer instead?

Here is the story of how they did it, explained simply:

1. The Problem: The "Empty Library"

Think of training a smart AI like teaching a student. If you want to teach them Luxembourgish, you need a library of books (text) and recordings (audio).

The Text Library: Full of questions and answers.
The Audio Library: Empty. There are no recordings of people asking these questions.

Usually, you'd have to hire actors to record thousands of hours of audio. That's expensive and slow.

2. The Solution: The "Robot Voice Factory"

Instead of hiring actors, the researchers built a Robot Voice Factory.

Step 1: They took existing text questions (from English) and translated them into Luxembourgish.
Step 2: They used different AI "Voice Actors" (TTS systems) to read these questions out loud.
Step 3: They paired these robot voices with the correct text answers.

Now, they had a massive "fake" library of spoken questions to train their AI.

3. The Experiment: Testing Different "Voice Actors"

The researchers didn't just use one robot voice. They tried five different types of TTS engines (like MMS-TTS, Qwen, and OmniVoice). Some were designed to clone a specific person's voice, while others were designed to create a brand-new, unique voice from scratch.

They also tried two main strategies:

The Single Voice: Training the AI on questions spoken by just one type of robot voice.
The Mixed Choir: Training the AI on a huge mix of questions spoken by many different robot voices (about 230,000 questions in total).

4. The Surprising Discovery: "Good Sounding" isn't "Good Learning"

Here is the most interesting part. The researchers had a "Sound Quality Meter" (a tool that scores how natural a robot voice sounds to a human ear).

Expectation: They thought the robot voices that sounded the most natural and human-like would make the best teachers.
Reality: The voices that sounded the best to human ears actually made the AI perform worse at answering questions.
The Winner: The AI learned best when it was trained on a diverse mix of voices, even if some of those voices sounded a bit more robotic or artificial.

The Analogy: Imagine learning to recognize a friend's face. If you only look at one perfect, high-definition photo, you might get confused when they wear a hat or stand in the dark. But if you look at a messy pile of photos—some blurry, some in black and white, some with different lighting—you learn to recognize the person much better. The AI needed the "messy pile" of diverse voices to learn the language, not just the "perfect" voice.

5. The Result: A Working System Without Real Humans

By using this "Mixed Choir" of robot voices, the researchers built a system that could listen to a spoken Luxembourgish question and give a text answer.

When they tested it on real human speakers (two different people), the system performed surprisingly well.
It proved that you don't need a massive library of real human recordings to build a spoken question-answering system for rare languages. You can build a very capable one using synthetic data, as long as you use the right kind of synthetic data.

The Bottom Line

This paper shows that for languages with few speakers, we don't need to wait for thousands of people to record themselves. We can use AI to generate the training data, but we must be careful: Quantity and variety matter more than perfection. A diverse mix of "imperfect" robot voices teaches the AI better than a single "perfect" robot voice.

Technical Summary: LuxSQA

Problem Statement

Spoken Question Answering (SQA) systems have historically relied on high-resource languages and carefully recorded speech corpora, limiting their applicability in low-resource settings. Traditional cascade approaches (ASR followed by text-based QA) suffer from error propagation when Automatic Speech Recognition (ASR) output is noisy and discard prosodic cues essential for answer localization. While recent speech-LLM architectures offer parameter-efficient solutions by connecting frozen speech encoders to Large Language Models (LLMs) via lightweight projectors, these systems still require substantial task-specific training data. For low-resource languages like Luxembourgish, the scarcity of human-recorded QA corpora presents a significant barrier to developing robust SQA systems. This paper investigates whether Text-to-Speech (TTS) augmentation can bridge this gap, specifically addressing whether synthetic speech can serve as effective training data without requiring large-scale human recordings, and how sensitive the resulting systems are to TTS quality, voice diversity, and corpus size.

Methodology

Data Construction Pipeline

The authors constructed a Luxembourgish SQA dataset by leveraging existing text-based resources:

Source Translation: English questions from the SQuAD v1.1 corpus were translated into Luxembourgish using the TranslateGemma 12B Instruct model.
Synthesis: The translated text questions were converted into spoken audio using multiple TTS systems:
- MMS-TTS: Meta's Massively Multilingual Speech TTS.
- Qwen3-TTS: Supporting CustomVoice (CV) cloning and VoiceDesign (VD) generation.
- OmniVoice: A zero-shot TTS model supporting over 600 languages, also offering CV and VD modes.
Corpora: The study utilized a single-source corpus of approximately 48k questions (SQuAD-LB-TTS) and a multi-source "4TTS mix" combining variants of Qwen3-TTS and OmniVoice to create a dataset of approximately 230k questions.
Evaluation Sets: Performance was evaluated on LLAMA-LB-Test, a 300-question set where Luxembourgish questions were human-translated and recorded by two native speakers (Speaker 1 and Speaker 2), providing real-world evaluation conditions.

Model Architecture

The system employs a parameter-efficient SLAM-style architecture:

Speech Encoder: A frozen Whisper-large-v3-turbo encoder processes the audio input.
Projector: A linear projector downsamples frame-level embeddings (by a factor of $k=5$ ) and maps them into the token embedding space of the LLM.
LLM Backend: Frozen multilingual LLMs (EuroLLM-9B or Apertus-8B) serve as the generation engine.
Adaptation: Only the projector and LoRA adapters (rank $r=16$ , scaling $\alpha=16$ ) are trained. The model is optimized to map spoken questions directly to textual answers using cross-entropy loss, operating in a speech instruction-following regime rather than pure transcription.

Training Strategies

The authors compared several training configurations:

Single-source: Training on data from one specific TTS engine (e.g., MMS-TTS, Qwen-VD, Omni-C).
Multi-source: Training on the combined 4TTS mix.
Initialization: Comparing training from scratch versus initialization from an ASR-adapted checkpoint (trained on LuxASR data).

Evaluation Metrics

Given the morphological richness of Luxembourgish and the tendency for paraphrasing, exact-match metrics were deemed insufficient. The paper utilizes two semantics-aware metrics:

Semantic Similarity (SemSim): Uses a multilingual sentence encoder to compute the maximum cosine similarity between the gold answer and any span in the prediction.
LLM Judge Mean: An average score from two instruction-tuned judges (GPT-4o and Gemini-2.5-Flash) evaluating the appropriateness of the answer on a scale of 0–1, accepting translations and variants.

Key Results

Performance Gains

Synthetic Data Efficacy: Models trained on speaker-specific synthetic Luxembourgish speech significantly outperformed generic Luxembourgish-adapted models. The best configuration (EuroLLM-9B with 4TTS mix) achieved 72.6% SemSim and 60.1% LLM Judge Mean for Speaker 1, and 74.8% SemSim and 61.8% LLM Judge Mean for Speaker 2.
Closing the Gap: These scores approached or exceeded the semantic similarity of the text-only baseline, demonstrating that carefully constructed synthetic data can compensate for the lack of real QA recordings.
Initialization Benefits: Initializing the projector from an ASR-adapted checkpoint further improved performance across both speakers compared to training from scratch.

TTS Quality vs. Downstream Performance

Non-Monotonic Correlation: A critical finding is that no-reference TTS quality scores (e.g., NISQA MOS, DNSMOS) do not monotonically predict downstream QA performance.
- Omni-C achieved the highest DNSMOS overall score (3.38) but yielded lower QA performance than the 4TTS mix.
- MMS-TTS received strong perceptual scores but remained among the weaker setups in QA tasks.
Importance of Diversity: Multi-source configurations and VoiceDesign-based setups (which generate diverse voices from text descriptions) consistently outperformed single-source or voice-cloning configurations, suggesting that voice diversity is more critical for robustness than raw audio naturalness scores.

Model Comparisons

Backbone: EuroLLM-9B generally outperformed Apertus-8B in higher-performing configurations.
Stability: High-performing configurations remained robust across both evaluation speakers, indicating that the training data configuration is the primary driver of success rather than speaker-specific randomness.

Significance and Claims

The paper positions LuxSQA as a methodological contribution rather than a model comparison, offering a complete workflow for constructing and exploiting TTS-augmented SQA data in low-resource languages.

Viability of TTS Augmentation: The study demonstrates that a usable SQA pipeline for low-resource languages can be built without collecting large human-recorded QA corpora. Speaker-specific synthetic training substantially improves performance on real spoken inputs.
Task-Specific Evaluation of Synthetic Data: The authors argue that synthetic speech must be evaluated as task-specific training data rather than solely as natural-sounding audio. High perceptual quality (MOS) is necessary but not sufficient; the synthetic data must preserve the lexical, phonetic, and prosodic cues required for the speech encoder and projector to recover question content reliably.
Persistent Speech-Text Gap: Despite the gains, a gap remains between text-only and speech-based performance. Even the best speaker-specific systems scored lower on LLM Judge Mean than the text baseline, suggesting that translation noise, pronunciation mismatches, and imperfect speech-to-LLM alignment still result in information loss.

The paper concludes that while TTS augmentation is a powerful tool for low-resource SQA, future efforts must balance manual and synthetic speech collection and focus on evaluating synthetic data based on its utility for the specific downstream task rather than generic audio quality metrics.