💬 NLP

BaltiVoice: A Speech Corpus and Fine-tuned Whisper ASR System for the Balti Language

This paper introduces BaltiVoice, the first publicly available speech corpus and fine-tuned Whisper ASR model for the Balti language, which significantly reduces word error rates from a 182.18% zero-shot baseline to 30.07% on a 16.8-hour dataset derived from Mozilla Common Voice.

Original authors: Muhammad Ali

Published 2026-06-03✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammad Ali

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a library of books, but for one specific language—Balti, spoken by about 400,000 people in Pakistan and India—there are no books at all. Not just no books, but no voice assistants, no dictation software, and no way for computers to understand the spoken word. It's like trying to navigate a city with no street signs or maps.

This paper introduces BaltiVoice, a project designed to build that first map.

The Problem: A Language in the Dark

Balti is a unique language with its own sounds and grammar, written in a beautiful script called Nastaliq (which looks like Urdu). Despite having a large community of speakers, it has been completely invisible to the world of Artificial Intelligence. If you tried to ask a smart computer to "listen" to Balti before this project, it would be like asking a dog to read a book; the computer would just guess randomly, getting almost everything wrong.

The Solution: Building a Training Gym

To teach a computer to speak a language, you need to show it thousands of examples of people speaking it. The author, Muhammad Ali, went to a massive online community project called Mozilla Common Voice. Think of this as a global recording booth where volunteers read sentences aloud.

The Collection: Ali gathered 16.8 hours of recorded speech.
The Volume: This equals 10,060 sentences spoken by 136 different people.
The Validation: Just like a teacher grading homework, other volunteers checked these recordings to make sure they were correct.

This collection is now called the BaltiVoice corpus. It's the first-ever public "textbook" for teaching computers about the Balti language.

The Teacher: Whisper and the "Urdu" Trick

The author didn't build a computer brain from scratch. Instead, he used a pre-existing, very smart AI model called Whisper (specifically the "small" version).

Imagine Whisper as a polyglot student who has already studied 99 languages (like English, Spanish, and Mandarin) for thousands of hours. However, this student has never heard Balti before. If you asked this student to listen to Balti right now, they would hallucinate nonsense, getting about 182% of the words wrong (which means they are inventing words that weren't even said).

To fix this, the author used a clever trick:

The Analogy: Since Balti is written in the Nastaliq script (which is very similar to Urdu), the author told the AI, "Hey, pretend this is Urdu for a moment."
The Training: The AI was then "fine-tuned." This is like taking that polyglot student and giving them a crash course using the 16.8 hours of Balti recordings. The student had to listen, read the text, and learn the specific sounds of Balti.

The Results: From Chaos to Clarity

After about 2 hours of training on a standard computer, the results were dramatic:

Before Training: The AI was guessing wildly (182% error rate). It was essentially making things up.
After Training: The AI's mistakes dropped to 30%.

What does a 30% error rate mean?
Imagine the AI listening to a sentence. If the sentence has 10 words, the AI will get about 7 right and 3 wrong.

Is it perfect? No. It's not good enough yet for a doctor's dictation or a legal transcript where every word must be exact.
Is it useful? Yes. It proves that the language can be understood by machines. It's the difference between a blind person stumbling in the dark and a person who can now see a faint light on the horizon.

Why This Matters

The paper emphasizes that this isn't just about getting a high score; it's about starting the conversation.

The Baseline: Before this, there was no way to measure progress. Now, researchers have a "starting line" to run from.
The Future: The author hopes this open-source "gym" (the data and the trained model) will allow other scientists to come in, do more training, and eventually lower that error rate.

The Bottom Line

This paper is a foundational step. It took a language that was invisible to AI, built a small library of spoken examples, and taught a smart computer how to listen to it. While the computer still makes mistakes (about one word in three), it has moved from "total confusion" to "understanding the basics," opening the door for future tools that could help Balti speakers interact with technology in their own language.

Technical Summary: BaltiVoice

Problem Statement
The Balti language (ISO 639-3: bft), spoken by approximately 400,000 people in Gilgit-Baltistan, Pakistan, and parts of Ladakh, India, has historically been absent from Natural Language Processing (NLP) and Automatic Speech Recognition (ASR) research. Despite being a Tibetic language with a distinct phonology and grammar, and written in a Nastaliq-based script adapted from Urdu, there were no publicly available ASR resources, annotated speech corpora, or baseline systems for the language. Consequently, speakers lack access to voice interfaces, dictation software, and accessibility tools in their native language, and researchers have no metric to measure progress in Balti ASR.

Methodology
To address this gap, the authors developed BaltiVoice, a pipeline for data collection, preprocessing, and model fine-tuning:

Dataset Construction: The corpus was derived from the Mozilla Common Voice Balti subset. The authors utilized 10,060 validated utterances (out of 10,547 recorded clips), totaling 16.8 hours of speech. The data consists of read-speech recordings in native Nastaliq script.
- Preprocessing: Audio files were converted from MP3 to 16 kHz mono WAV format. Utterances with fewer than two words were filtered out.
- Splitting: A strict speaker-disjoint split was applied using GroupShuffleSplit (seed 42) to ensure no overlap between speakers in the training and validation sets. This resulted in 9,519 training utterances (122 speakers) and 538 validation utterances (14 speakers).
- Normalization: No text normalization was applied; punctuation was preserved as provided. The authors noted a limitation regarding Unicode ambiguity in the Nastaliq script (e.g., differing codepoints for visually identical characters) but deferred systematic normalization to future work.
Model Architecture and Training:
- Base Model: OpenAI's Whisper-small (244M parameters) was selected as the base model. It was chosen over larger variants (e.g., Whisper-medium) due to memory constraints on the NVIDIA T4 GPU used for training.
- Tokenization: The tokenizer was initialized with language="urdu" and task="transcribe". This choice was motivated by the script similarity between Balti and Urdu (both use Nastaliq), allowing the model to handle Balti Unicode characters correctly without character loss during round-trip tokenization.
- Fine-tuning: The model was fine-tuned using HuggingFace Transformers' Seq2SeqTrainer with AdamW optimizer, a learning rate of $1 \times 10^{-5}$ , and fp16 precision. Training ran for 1,000 steps on 16.8 hours of data, with checkpoints saved every 250 steps.

Key Contributions
The paper presents three primary artifacts, all released publicly on HuggingFace and GitHub:

BaltiVoice Corpus: A 16.8-hour, 10,060-utterance read-speech corpus with native Nastaliq transcriptions, released under CC0.
Whisper-small-balti: A fine-tuned ASR model specifically for the Balti language.
Reproducible Pipeline: Complete training code, a Colab notebook, and a live Gradio demo for transcription.

Results
The fine-tuned model demonstrated significant improvement over the zero-shot baseline:

Zero-Shot Baseline: When applied to Balti without fine-tuning, Whisper-small produced a Word Error Rate (WER) of 182.18%. The authors note that a WER above 100% indicates the model is hallucinating words not present in the reference, confirming Balti falls entirely outside the model's pretraining distribution.
Fine-Tuned Performance: After 1,000 training steps, the model achieved a WER of 30.07% on the held-out validation set.
Error Analysis: Qualitative analysis suggests that most errors are single-character substitutions at word endings, consistent with the model learning lexical patterns but struggling with the morphological complexities of the agglutinative language. Whole-word deletion or insertion errors were less frequent.

Significance and Claims
The authors frame the significance of this work as establishing a measurable starting point for a language that previously had none.

Baseline Establishment: The primary goal is to provide a reproducible baseline to accelerate future research in Balti NLP.
Feasibility of Low-Resource Transfer: The reduction of WER from 182% to 30% using only 16.8 hours of data suggests that cross-lingual transfer from related languages (specifically Urdu and Tibetan, which share script and phonological features) is effective even for languages absent from the pretraining distribution.
Modest Expectations: The authors explicitly state that a 30% WER is "too high for dictation" or general accessibility tools, as it requires correcting roughly one in three words. However, they argue the output may be usable for narrower tasks like keyword spotting or topic detection where exact transcription is less critical.
Future Directions: The paper identifies clear paths for improvement, including text normalization for Balti morphology, expanding the corpus with spontaneous (conversational) speech, and experimenting with larger model variants (e.g., Whisper-medium) under greater compute budgets.

The paper concludes by emphasizing the release of all artifacts to lower the barrier for future research, while acknowledging limitations such as the use of read-speech data rather than spontaneous conversation.