💬 NLP

From a Multilingual Streaming ASR Backbone to Kenyan-Language Systems: Data-Centric Adaptation of Nemotron 3.5 for Kikuyu, Dholuo, and Kalenjin

This paper presents a comprehensive engineering study on adapting NVIDIA's Nemotron 3.5 streaming ASR model to Kikuyu, Dholuo, and Kalenjin languages through data-centric strategies like corpus auditing and normalization, achieving specific WER and CER metrics while transparently reporting challenges, negative findings, and the absence of state-of-the-art claims due to non-standard evaluation protocols.

Original authors: Mark Gatere

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mark Gatere

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to listen to the world. This robot, let's call him "Nemotron," is already a whiz at understanding many languages, but he's mostly used to the big, popular ones. Now, imagine you want him to learn three specific languages spoken in Kenya: Kikuyu, Dholuo, and Kalenjin. This is the world of Automatic Speech Recognition (ASR). Think of ASR as the robot's "ear" and "brain" working together to turn sound waves into written words.

But here's the tricky part: teaching a robot a new language isn't just about playing it a few songs. It's like trying to teach a chef a new recipe when the ingredients are messy, the instructions are written in a code you don't fully understand, and some of the kitchen tools are missing. In the world of AI, this is called the "low-resource" problem. It doesn't just mean there isn't enough audio; it means the audio might have weird spelling mistakes, missing parts, or confusing background noises that make it hard for the robot to learn the difference between a real word and a glitch.

This paper is a story about how a team of engineers took that super-smart robot, gave him a special "bridge" lesson using a related language, and then tried to teach him these three Kenyan languages. They didn't just want him to understand the words; they wanted him to listen in real-time, like a human chatting on a phone call, rather than waiting until the whole conversation is over to write it down. It's a tale of data detective work, fixing broken instructions, and seeing how far a robot can go when you give it the right tools and the right amount of patience.

The Story of the Kenyan Language Project

The paper, written by Mark Gatere from C-elo Labs, is essentially an engineering diary. It documents the journey of adapting a massive, pre-trained AI model (Nemotron 3.5) to understand Kikuyu, Dholuo, and Kalenjin. The team didn't start from scratch; they started with a "bridge." They took a version of the robot that had already learned Kenyan Swahili and used that as a starting point. Think of it like learning Italian if you already know Spanish: the grammar and vocabulary are close enough that you have a huge head start, even if the two languages aren't identical twins.

The Detective Work: Cleaning the Data
Before the robot could learn, the team had to act as data detectives. The raw audio files they found were messy. Some had missing audio, some had transcripts with weird symbols (like "long pause" written out as words instead of silence), and some had spelling that didn't match the actual sounds.

The "Delete vs. Fix" Dilemma: The team faced a tough choice. If a transcript had a weird symbol, should they try to guess what it meant and fix it, or just throw the whole recording away? They decided that guessing was too dangerous. If they weren't 100% sure what the speaker said, they deleted the row. It's like a chef throwing away a cake if they aren't sure if the ingredient was sugar or salt, rather than risking a bad taste. This meant they lost some data, but the data they kept was trustworthy.
The "Marker" Problem: In the Kalenjin data, they found that the recordings had little notes written in the text like [pause] or [cs] (code-switch). The robot was trying to learn to say "pause" out loud! The team had to scrub these notes out completely so the robot learned to recognize silence and language switching, not the words describing them.

The Training: A Marathon, Not a Sprint
Once the data was clean, they started the training. They didn't just train the robot once; they did it in stages, like leveling up in a video game.

Full-Parameter Tuning: Instead of just tweaking a few settings, they let the robot relearn almost everything about how it processes sound for these specific languages.
Streaming Mode: Crucially, they kept the robot in "streaming" mode. This means the robot has to guess the words as the sound comes in, chunk by chunk, without waiting for the end of the sentence. It's like trying to transcribe a live radio broadcast versus reading a finished book. This is much harder but much more useful for real-life apps.

The Results: How Well Did They Do?
The paper is very honest about what worked and what didn't. It doesn't claim to have "solved" these languages, but it shows massive progress.

Kikuyu: The robot got really good. After all the cleaning and training, it achieved a Word Error Rate (WER) of 42.97%. In plain English, this means if the robot heard 100 words, it got about 57 of them right. That's a huge improvement from where they started. They also measured a "No-Space Character Error Rate" of 7.79%, which is a fancy way of saying that even if it got the word boundaries wrong (like writing "in ya" instead of "inya"), the actual letters were mostly correct.
Dholuo: This one performed even better, with a WER of 33.98%. The robot made fewer mistakes here, getting about two-thirds of the words right.
Kalenjin: This one is still a "work in progress." The robot got a WER of 68.74% on a filtered test set. This is a high error rate, meaning the robot is still struggling. The authors explain that this score is a "diagnostic" result, not a final one, because they had to be very strict about what data they tested on (removing anything with numbers or short sentences). They are still figuring out the best way to teach this language.

What the Paper Doesn't Claim
The authors are very careful not to overhype their results.

No "State-of-the-Art" Claim: They explicitly say they are not the best in the world yet. They haven't compared their robot to every other robot on the planet using the exact same test. Their numbers are internal, meaning they are great for tracking their own progress, but they aren't a final scorecard for the whole world.
The "Bridge" Mystery: They used a Kenyan Swahili model as a starting point, but they admit they didn't prove that this was better than starting from the raw, untrained robot. It was a smart guess that worked, but they didn't run a side-by-side test to prove it was the only way.
The "Repeated" Test: They used the same test data over and over to guide their decisions. While they didn't let the robot see the answers during training, they looked at the results to decide when to stop. This means the scores might be slightly optimistic because the team got to know the test questions too well.

The Real-World Test
The team didn't just stop at numbers. They built a prototype system where a user could talk to the robot through a web browser, and the robot would type back in real-time. They made sure the robot was isolated so that if the Dholyu version broke, it wouldn't crash the Kikuyu version. They also set up security so that only logged-in users could access the service, protecting the robot's "brain" from being stolen.

The Takeaway
This paper is a masterclass in "data-centric" AI. It shows that having a powerful robot model isn't enough; you have to be a meticulous editor of the data you feed it. By cleaning up the transcripts, removing the confusing "notes," and training the robot to listen in real-time, they turned a general-purpose AI into a specialized tool for Kenyan languages.

For Kikuyu and Dholuo, they have a working, live system that is good enough to be useful. For Kalenjin, they have a roadmap of what's still broken and how to fix it. The biggest lesson isn't just about these three languages; it's that for low-resource languages, the secret sauce isn't just bigger computers—it's better data hygiene and a careful, step-by-step engineering process. The robot is learning, but it's learning because humans did the hard work of cleaning up the mess first.

Technical Summary: Data-Centric Adaptation of Nemotron 3.5 for Kenyan Languages

Problem Statement

Automatic Speech Recognition (ASR) for African languages faces constraints beyond simple data scarcity. The paper identifies a broader set of operational challenges: orthographic inconsistency, annotation artifacts (e.g., non-speech markers treated as text), missing audio objects, speaker and domain imbalances, and evaluation procedures that diverge from deployment realities. Specifically, the authors address the need to adapt a multilingual streaming ASR backbone to Kikuyu, Dholuo, and Kalenjin while maintaining true streaming capabilities (low-latency, stateful inference) rather than relying on offline encoder-decoder models. The study questions whether a large multilingual model can be successfully adapted, evaluated, and served as a streaming system under realistic data curation and infrastructure constraints.

Methodology

System Architecture and Base Model

The study utilizes NVIDIA Nemotron 3.5 ASR Streaming 0.6B as the foundation. The architecture is a FastConformer RNN-T (Recurrent Neural Network Transducer) featuring:

A 24-layer cache-aware encoder (dimension 1024).
A 128-dimensional one-hot language identifier prompt.
A SentencePiece vocabulary of 13,087 units.
Streaming Mechanism: The model uses cache-aware attention with a context configuration of [56, 13] (56 left-context units, 13 right-context units), enabling stateful inference across audio chunks without full-prefix recomputation.

Adaptation Strategy

Instead of adapting directly from the raw NVIDIA base, the authors initialize from a Kenyan Swahili-adapted bridge checkpoint (publicly released by Tony Kipkemboi). This "bridge" is treated as an initialization point to leverage shared acoustic environments, names, and code-switching patterns common in Kenyan speech, rather than as a lexical teacher.

Fine-Tuning: Full-parameter fine-tuning is performed on target-language audio-transcript pairs.
Prompting: The language prompt remains sw-KE (Swahili-Kenya) throughout training; the target language is learned via the supervised data and parameter updates, not by changing the prompt.
Training Protocol: The process involves staged continuation, where a selected checkpoint from one stage initializes the next. Hyperparameters include AdamW optimization with a low learning rate ( $5 \times 10^{-7}$ ) and duration-based batching.

Data Curation and Governance

A significant portion of the work focuses on data forensics and language-specific curation using African Next Voices (ANV) datasets.

Pipeline: A deterministic pipeline handles audio decoding, duration filtering, and text normalization. Rows with missing audio, empty transcripts, or unresolved Unicode replacement characters are discarded.
Language-Specific Decisions:
- Kikuyu: Strict orthographic normalization for contrastive vowels (e.g., ˜ı, ˜u), rejecting rows with uncertain character mappings.
- Dholuo: Retained a "frozen historical" label policy where the token cs (code-switch marker) was not removed from references, allowing for controlled comparison within that specific lineage.
- Kalenjin: Iterative cleaning from clean-v1 to clean-v3. This involved removing annotation wrappers ([cs], (pause)) and, in clean-v3, excluding rows containing literal phrases like "long pause," digits, or fewer than three tokens to prevent short-utterance over-generation.
Split Integrity: The authors emphasize strict separation of train, validation, and test sets. A correction was applied to Kalenjin splits to ensure the validation set did not contain test-origin rows, a flaw present in earlier experimental runs.

Evaluation Protocol

The study distinguishes between trainer validation (full-utterance offline evaluation used for checkpoint selection) and true-streaming evaluation.

True-Streaming: The internal evaluator mimics deployment by processing audio in bounded chunks, carrying encoder caches and decoder state, and emitting partial hypotheses.
Metrics: Word Error Rate (WER), Character Error Rate (CER), and No-Space CER (NS-CER), which removes spaces to account for morphological boundary inconsistencies in agglutinative languages.
Held-Out Status: The test sets were never used for gradient updates but were consulted adaptively during model selection. The authors explicitly state these are internal diagnostics, not pristine one-shot benchmarks.

Key Results

Kikuyu

Status: Completed engineering case study with a live streaming deployment.
Performance: The champion model (v13) achieved 42.97% WER and 7.79% No-Space CER on the internal true-streaming test set.
Trajectory: Significant improvements were observed through staged continuation and orthographic curation, reducing WER by over 10 percentage points from the initial filtered baseline.

Dholuo

Status: Completed engineering case study with a live streaming deployment.
Performance: The champion model (v7) achieved 33.98% WER, 9.59% CER, and 8.13% No-Space CER.
Note: These results are under a "frozen historical" policy where the cs token remained in references. The authors caution that these scores are not directly comparable to systems using a different normalization policy.

Kalenjin

Status: Ongoing research case study; not yet a production system.
Performance: The v1-v checkpoint achieved 68.74% WER on a filtered clean-v3 diagnostic subset.
Caveats: This result is an "adaptive diagnostic" because the checkpoint selection used a mixed-source validation manifest containing test-origin rows. The high error rate reflects residual issues with corpus quality, boundary sensitivity, and short-utterance over-generation.

Significance and Claims

The paper makes no claim of state-of-the-art (SOTA) performance against public leaderboards (e.g., PazaBench). The authors explicitly state that internal results differ from public benchmarks in audio, normalization, and evaluation protocols.

Instead, the paper's significance lies in providing an auditable, end-to-end engineering account of adapting a multilingual streaming model to low-resource languages. Key contributions include:

Data-Centric Adaptation: Demonstrating that rigorous data curation (removing artifacts, fixing orthography, ensuring split integrity) is as critical as model architecture for low-resource success.
Streaming Fidelity: Proving that a cache-aware RNN-T can be adapted from a bridge checkpoint to target languages while preserving true streaming behavior from training through deployment.
Negative Evidence: Reporting failures and limitations (e.g., Kalenjin's mixed-source selection flaw, short-utterance over-generation, and the insufficiency of aggregate WER) to prevent future work from repeating these errors.
Reproducibility Framework: Defining a workflow for artifact preservation, checkpoint promotion, and isolated language service deployment, including specific protocols for handling "negative" results and data governance.

The work concludes that low-resource ASR is a problem of valid supervision, evaluation discipline, and deployment fidelity as much as it is a problem of model scale. It offers a reproducible pathway for moving from research experiments to interactive speech technology without abandoning scientific caution.