SloPal: A 60-Million-Word Slovak Parliamentary Corpus with Aligned Speech and Fine-Tuned ASR Models

This paper introduces SloPal, a comprehensive 60-million-word Slovak parliamentary corpus with aligned speech data and fine-tuned Whisper models that significantly improve automatic speech recognition performance for this low-resource language.

The Gist

Imagine you are trying to teach a brilliant, multilingual robot (let's call him "Whisper") how to speak Slovak. The problem? Whisper has read millions of books in English, French, and German, but he has barely heard a single word of Slovak. He's like a world-class chef who knows how to make a perfect soufflé but has never seen a potato.

This paper introduces SloPal, a massive project designed to give Whisper a crash course in Slovak by feeding him the most formal, structured, and abundant source of the language available: Parliamentary speeches.

Here is the story of how they did it, broken down into simple concepts:

1. The Problem: The "Low-Resource" Desert

In the world of AI, some languages are "rich" (like English) with mountains of data, while others are "poor" (low-resource). Slovak is in the latter category. Before this project, there were fewer than 100 hours of public Slovak audio available to train AI. That's like trying to learn a language by listening to a single 15-minute podcast. The AI's performance was terrible, making mistakes on almost every word.

2. The Solution: The "Parliamentary Goldmine"

The researchers realized that the Slovak Parliament (the National Council) has been recording and transcribing every speech since 2001. It's a treasure trove!

• The Text: They gathered 66 million words of transcripts. That's like filling a library with thousands of books, all written by politicians.
• The Audio: They found the actual recordings of these speeches.
• The Match: They combined the audio with the text to create a "study guide" where the AI can hear a word and immediately see how it's spelled.

3. The Challenge: The "Mismatched Puzzle"

You might think, "Just download the audio and the text, and you're done!" But it wasn't that simple.

• The Problem: The official transcripts were huge blocks of text (sometimes hours long), while the audio files were messy. The timestamps (the "when" a word was spoken) in the official records were often wrong or missing. It was like having a script for a play and a recording of the play, but the script didn't say who was speaking or when.
• The Fix: They built a clever "translator" system. They used the AI to generate a rough draft of what was said, then used a smart matching algorithm to find "anchors" (common words that appear in both the rough draft and the official text). Once they found these anchors, they could stitch the audio and text together perfectly, cutting them into bite-sized 30-second chunks (the perfect size for the AI to digest).

4. The Result: The "Whisper" Workout

Once they had this massive, clean dataset (called SloPalSpeech), they put the AI through a rigorous training camp.

• Before Training: The AI was clumsy, making mistakes on about 33% of the words it heard in Slovak.
• After Training: The AI became a pro. They fine-tuned different sizes of the AI, and the results were shocking.
  • The Small Model (which is tiny and fast, like a smartphone app) improved so much that it started performing almost as well as the Huge Model (which is massive and slow, like a supercomputer).
  • They reduced the error rate by up to 70%.

5. The Analogy: The "App vs. The Library"

Think of the AI models like students:

• The Base Large Model is a genius student who has read every book in the world but doesn't know Slovak.
• The Small Model is a smart student who knows a little bit of everything but needs to specialize.
• SloPal is the specialized textbook.
• The Magic: By studying this specific textbook, the "Small Student" learned Slovak so well that he could beat the "Genius Student" at Slovak tasks, even though the Genius has a much bigger brain. This means we can now run high-quality Slovak speech recognition on regular phones, not just giant servers.

6. Why This Matters

The researchers didn't just keep this to themselves. They released everything for free:

1. The Text: All 66 million words for anyone to study politics, history, or language.
2. The Audio: The aligned recordings for anyone building speech tech.
3. The Models: The "trained" AI brains that can now understand Slovak speech with high accuracy.

In a nutshell: This paper took a messy, underutilized government archive, cleaned it up, and used it to teach a robot how to speak Slovak fluently. They proved that you don't need a supercomputer to do this; you just need the right data and a smart way to organize it. Now, Slovak speakers can finally talk to their devices, and researchers have a massive new tool to study the language.

Technical Summary

1. Problem Statement

Automatic Speech Recognition (ASR) for Slovak remains a significant challenge due to its classification as a low-resource language.

• Data Scarcity: Publicly available training data for Slovak is limited to fewer than 100 hours (e.g., Common Voice, FLEURS, VoxPopuli), which is insufficient for training high-performance models compared to high-resource languages like English (hundreds of thousands of hours).
• Performance Gap: State-of-the-art models like OpenAI's Whisper exhibit high Word Error Rates (WER) on Slovak (e.g., ~33% for Whisper Small) compared to high-resource languages (<6%).
• Lack of Specialized Resources: Existing parliamentary corpora for other Slavic languages (Czech, Polish, Croatian) exist, but a comprehensive, speaker-segmented Slovak resource with aligned audio was missing.

2. Methodology

The authors developed a pipeline to construct a large-scale corpus and fine-tune ASR models through four main stages:

A. Data Collection

• Source: Data was harvested from MediaPortál (Slovak National Council's official portal) for audio and the Joint Czech and Slovak Digital Parliamentary Library for transcripts.
• Scale: The team collected 4,096 hours of audio and 2,199 transcript files (DOCX) covering sessions from 2001 to 2024.
• Validity Check: Initial alignment revealed a mismatch: audio recordings only covered forenoon sessions, while transcripts covered full days. The authors filtered transcripts to match the audio duration, ensuring valid ground truth.

B. Text Parsing and Segmentation

• Format Conversion: DOCX transcripts were converted to XHTML using Apache Tika.
• Speaker Identification: A heuristic-based approach was used to identify speaker annotations (bold text containing 1–3 names and <15 words). This was validated against a scraped list of National Council members.
• Cleaning: Non-verbatim content (transcriber notes in parentheses/brackets) was removed.
• Output: This process yielded 330,000 speaker-segmented transcripts containing 66 million words (SloPal Text Corpus).

C. Audio-Text Alignment (SloPalSpeech)

Since standard forced alignment failed due to long audio durations and transcript discrepancies, the authors developed a language-agnostic anchor-based alignment pipeline:

1. Reference Generation: WhisperX was used to generate a preliminary transcript with word-level timestamps from the audio (30-second chunks).
2. Anchor Selection: "Anchors" (reliable words) were identified by matching the reference transcript against the ground-truth transcript using Levenshtein distance (≤1) and context matching (4 words before/after).
3. Segmentation: Anchors were used to slice the ground-truth text and corresponding audio into segments ≤30 seconds, optimized for Whisper's architecture.
4. Quality Filtering: Segments were re-evaluated using Whisper. A WER threshold of 40% was applied to filter out misaligned or low-quality segments.

• Result: A final aligned dataset of 2,806 hours (SloPalSpeech), comprising 402,966 segments.

D. Model Fine-Tuning

• Models: Four variants of OpenAI's Whisper were fine-tuned: Small, Medium, Turbo (v3), and Large-v3.
• Training Strategy:
  • Used Hugging Face Transformers with early stopping (max 3 epochs).
  • Hyperparameters: Learning rates were adjusted (40× smaller than pre-training for smaller models; 1×10⁻⁵ for Large-v3). Weight decay (0.01) was introduced for Large-v3 to prevent overfitting.
  • Hardware: Multi-GPU training (NVIDIA A10) using Fully Sharded Data Parallel (FSDP) for the Large-v3 model.
• Evaluation: Models were tested on Common Voice 21 and FLEURS (Slovak subsets).

3. Key Contributions

1. SloPal Text Corpus: The largest Slovak parliamentary text resource to date, featuring 330,000 speaker-segmented transcripts (66M words) with rich metadata (speaker names, roles, session info).
2. SloPalSpeech: The first dedicated Slovak aligned speech dataset (2,806 hours) with 30-second segments, specifically optimized for ASR training.
3. Alignment Pipeline: A novel, language-agnostic anchor-based alignment method that bypasses the need for language-specific forced alignment models, applicable to any language with parliamentary transcripts.
4. Fine-Tuned Models: Four publicly released Whisper models fine-tuned on Slovak parliamentary data.

4. Results

The fine-tuning process yielded substantial improvements in Word Error Rate (WER):

Model Variant	Base WER (FLEURS)	Fine-Tuned WER (FLEURS)	Improvement
Whisper Small	36.1%	10.6%	−25.5 pp (70% reduction)
Whisper Medium	18.7%	7.6%	−11.1 pp
Whisper Large-v3	9.2%	5.5%	−3.7 pp

• Efficiency: The fine-tuned Whisper Small (244M parameters) achieved performance comparable to the base Whisper Large-v3 (1.5B parameters) but with 6× fewer parameters.
• Benchmark Comparison: The fine-tuned Large-v3 model outperformed NVIDIA's Canary-1B-v2 on the FLEURS benchmark (5.5% vs 6.82% WER), despite the NVIDIA model having been trained on FLEURS data.
• Generalization: Models showed improved performance not just on parliamentary data but also on general speech datasets (Common Voice), indicating the data did not cause catastrophic forgetting.

5. Significance

• Low-Resource ASR Advancement: This work demonstrates that domain-specific data (parliamentary speech) can drastically improve ASR performance for low-resource languages, narrowing the gap with high-resource languages.
• Resource Accessibility: By releasing the text corpus, aligned audio, and models under open licenses (CC BY 4.0 and MIT), the authors provide the most comprehensive open resource for Slovak NLP, ASR, and computational social science.
• Scalability: The anchor-based alignment pipeline offers a blueprint for creating similar corpora for other Slavic and low-resource languages where parliamentary records exist but lack aligned audio.
• Practical Deployment: The results suggest that smaller, fine-tuned models (like Whisper Small) are viable for production deployment in Slovak, offering near-large-model accuracy with significantly lower computational costs.