💬 NLP

FLEURS-Kobani: Extending the FLEURS Dataset for Northern Kurdish

This paper introduces FLEURS-Kobani, the first public Northern Kurdish benchmark consisting of 5,162 validated utterances recorded by 31 native speakers, which extends the FLEURS dataset to enable evaluation of automatic speech recognition and speech translation tasks for this under-resourced language.

Original authors: Daban Q. Jaff, Mohammad Mohammadamini

Published 2026-04-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daban Q. Jaff, Mohammad Mohammadamini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of language technology as a massive, high-tech library. For years, this library has been stocked with books (data) for over 100 languages, allowing computers to learn how to listen, speak, and translate. This library is called FLEURS. It's like a universal training gym where AI models go to get fit for tasks like recognizing speech or translating it.

However, there was a glaring hole in the library: Northern Kurdish (spoken by millions in Turkey, Syria, Iraq, and Iran) had no books. It was like a gym with a brand-new, empty weight rack for a specific group of people. Without data, computers couldn't learn to understand or speak this language effectively.

This paper introduces FLEURS-Kobani, a project designed to fill that empty rack.

🎙️ The Recording Studio: Building the Library

The authors didn't just copy-paste existing data; they built a new collection from scratch.

The Cast: They recruited 31 native speakers (mostly women, with a few men) from the University of Kobani. Think of them as the "actors" in a radio play.
The Script: They used 2,000 unique sentences from a standard text collection (FLORES) and asked the actors to read them aloud.
The Reality Check: Recording wasn't easy. Due to internet instability and the use of personal cell phones, the audio quality varied wildly. It was like trying to record a symphony in a room where the power keeps flickering and some musicians are using cheap microphones.
The Cleanup: Out of nearly 8,000 recordings, they had to throw away about 35% because of background noise, cut-off words, or people reading the wrong sentences. The remaining 5,162 clean recordings (about 18 hours of audio) became the new dataset.

🧪 The Test Drive: Putting AI to Work

Once the "library" was built, the authors needed to see if it actually helped computers learn. They used a famous AI model called Whisper (think of it as a very smart, but currently untrained, student) and gave it a crash course using their new data.

They ran three different training scenarios:

The Solo Student: The AI tried to learn only from the new Kobani data. It did okay, but struggled.
The Transfer Student: The AI first learned from a different Kurdish dataset (Common Voice) and then moved to Kobani. This was better.
The Two-Step Dance (The Winner): The AI first learned the basics on the Common Voice data, then refined its skills on the new Kobani data. This worked best. It's like learning to drive on a quiet practice track before hitting the busy highway. The result was a significant drop in errors, meaning the computer could finally understand Northern Kurdish speech much more accurately.

🌍 The Translation Challenge

The team also tested if the AI could listen to Northern Kurdish and instantly translate it into English (and other languages like French or Dutch).

Direct Translation: The AI tried to go straight from Kurdish to English. It got a decent score, but not perfect.
The "Pivot" Problem: They tried translating Kurdish to English, then English to French (a "pivot" strategy). Interestingly, this worked well for languages close to English (like French or Dutch) but failed for languages that are linguistically similar to Kurdish (like Central Kurdish or Persian). It's like trying to translate a joke: sometimes translating it through a common language (English) loses the specific cultural nuance needed for a similar language.

🚀 Why This Matters

Before this paper, Northern Kurdish was a "ghost language" in the world of speech technology—spoken by millions but invisible to computers.

The First Benchmark: This is the first time anyone has created a standardized "test" for Northern Kurdish speech. It's like finally giving a driver's license exam to a group that was previously just guessing how to drive.
Open Source: The data is now free for anyone to use (under a Creative Commons license), lowering the barrier for researchers to build better tools for Kurdish speakers.
Future Growth: The authors admit the data isn't perfect (mostly female speakers, some technical glitches), but it's a solid foundation. It's the first brick in a new building for Kurdish language technology.

In short: The authors built a new, specialized training ground for computers to learn Northern Kurdish. By recording, cleaning, and testing this data, they've given the AI a fighting chance to understand and speak a language that was previously left out of the conversation.

1. Problem Statement

The paper addresses the critical lack of standardized speech resources for Northern Kurdish (ISO 639-3: KMR), a widely spoken Indo-European language with over 30 million native speakers. While the FLEURS benchmark provides parallel speech data for over 100 languages to facilitate few-shot learning in Automatic Speech Recognition (ASR), Speech-to-Text Translation (S2TT), and Speech-to-Speech Translation (S2ST), it notably excludes Northern Kurdish.

Current Gap: Existing Kurdish speech resources are heavily skewed toward Central Kurdish (CKB). Northern Kurdish lacks high-quality, standardized evaluation benchmarks, hindering fair comparison and progress in ASR and translation tasks for this specific dialect.
Motivation: The authors aim to fill this resource gap by creating a dedicated, multi-task benchmark that aligns with the FLEURS methodology, enabling rigorous evaluation of speech technologies for Northern Kurdish.

2. Methodology

Data Collection and Recording

Source Material: The dataset is based on the Northern Kurdish portion of the FLORES text corpus (2,000 unique sentences), ensuring alignment with existing multilingual benchmarks.
Recording Platform: Data was recorded using the Jiridastkrd platform at Kobani University over two months (April–June 2025).
Participants: 31 native speakers (26 female, 5 male) participated. The gender imbalance was a result of socio-political constraints and the initial recruitment of female university students.
Procedure: Speakers were trained to read prompts on the platform. The process involved recruitment, outreach to diversify speakers, and specific training on recording protocols.

Data Validation and Curation

A rigorous two-step validation process was applied to ensure data quality:

Prompt Conformity: Checking if the speaker read the intended sentence without significant deviation.
Audio Quality: Assessing intelligibility, background noise, and recording levels.

Exclusion Criteria: Recordings were discarded for prompt mismatches, truncation, unauthorized insertions, pronunciation errors (specifically for foreign names/abbreviations where no single "correct" reading exists), excessive noise, or low volume.
Outcome: Out of 7,897 initial recordings, 5,162 were validated as high quality (65.5%), resulting in 18 hours and 24 minutes of speech data.

Data Partitioning

The validated utterances were split into Train, Development (Dev), and Test sets based on their alignment with the English FLEURS dataset:

Train: 3,870 utterances (13h 47m).
Dev: 491 utterances (1h 42m) – Note: This includes 149 non-matching samples added to increase size.
Test: 801 utterances (2h 55m).
License: The dataset is released under CC BY 4.0.

Experimental Setup

The authors established baselines using the Whisper v3-large model:

ASR: Fine-tuned under three conditions:
1. Common Voice (CV) only.
2. FLEURS-Kobani (F-K) only.
3. Two-stage fine-tuning: CV $\rightarrow$ F-K.
End-to-End S2TT (KMR $\rightarrow$ EN): Fine-tuned Whisper directly on the F-K training set.
Pivot-based S2TT: Generated parallel data for KMR $\rightarrow$ French, Dutch, Central Kurdish, and Persian using English as a pivot.
Cascaded S2TT: Transcribed audio via ASR (Whisper) and then translated via Machine Translation (NLLB-1.3B).

3. Key Contributions

FLEURS-Kobani Dataset: The first public, standardized speech benchmark for Northern Kurdish, comprising 5,162 validated utterances across three splits.
Multi-Task Benchmarking: The dataset supports evaluation for ASR, S2TT, and S2ST, filling a void in the FLEURS suite for the Kurdish language family.
Baseline Performance: Comprehensive baseline results using Whisper v3-large, demonstrating the efficacy of transfer learning strategies (specifically the two-stage fine-tuning approach).
Resource Accessibility: The dataset is publicly available on Hugging Face, lowering the barrier for future research into low-resource Kurdish speech technologies.

4. Results

Automatic Speech Recognition (ASR)

The two-stage fine-tuning strategy (Common Voice $\rightarrow$ FLEURS-Kobani) yielded the best performance, significantly outperforming single-stage training.

Word Error Rate (WER): 28.11% (Test set).
Character Error Rate (CER): 9.84% (Test set).
Comparison: Training on FLEURS-Kobani alone resulted in a WER of 38.09%, while Common Voice alone yielded 37.02%. The combined approach reduced WER by nearly 10 percentage points.

End-to-End Speech-to-Text Translation (S2TT)

KMR $\rightarrow$ EN: The Whisper model achieved 8.68 BLEU on the test set.
Pivot Languages: Translations to French (FR) and Dutch (NL) performed better than to Central Kurdish (CKB) or Persian (FA), likely due to the pivot language (English) being closer to FR/NL typologically, whereas lexical/typological divergences hindered KMR-CKB performance despite linguistic similarity.

Cascaded S2TT

Pipeline: ASR (Whisper) + MT (NLLB).
Performance: The cascaded approach achieved 19.60 BLEU on the test set, compared to a baseline of 27.36 BLEU (direct translation of references). This highlights the error propagation inherent in cascaded systems but establishes a functional baseline for future end-to-end model development.

5. Significance and Future Work

Language Inclusion: This work is a pivotal step toward digital inclusion for Northern Kurdish speakers, providing the first dedicated benchmark for ASR and translation tasks.
Research Acceleration: By releasing the dataset and baselines, the authors enable the community to benchmark new models and improve upon the current Whisper baselines.
Challenges Identified: The paper highlights specific challenges, including the impact of limited formal education in Kurdish among speakers (affecting reading fluency), socio-political constraints on data collection, and the difficulty of standardizing pronunciation for foreign proper names.
Future Directions: The authors suggest future research should focus on:
- Speaker-level granular evaluation to analyze fluency and dialect variability.
- Expanding the dataset to cover additional Kurdish varieties (e.g., Southern Kurdish, Hewrami).
- Developing models specifically robust to the dialect continuum nature of Kurdish.

In conclusion, FLEURS-Kobani successfully bridges a major resource gap for Northern Kurdish, providing a robust foundation for advancing automatic speech recognition and translation technologies in a low-resource, high-impact language context.