Bolbosh: Script-Aware Flow Matching for Kashmiri Text-to-Speech

This paper introduces Bolbosh, the first open-source neural Text-to-Speech system for Kashmiri, which utilizes a script-aware, supervised cross-lingual adaptation strategy based on Optimal Transport Conditional Flow Matching and a three-stage acoustic enhancement pipeline to overcome the limitations of zero-shot multilingual baselines and achieve significantly higher speech quality and intelligibility.

The Gist

Imagine you have a beautiful, ancient library filled with stories written in a very specific, intricate script. This library belongs to the Kashmiri language, spoken by about 7 million people. For a long time, the "robots" (computer programs) that read these stories aloud have been terrible at it. They either mumbled the words, got the vowels wrong, or sounded like a broken radio.

The paper you shared introduces a new robot named Bolbosh (which means "voice" in Kashmiri). It's the first robot specifically trained to read Kashmiri stories out loud with clarity and emotion.

Here is how they built it, explained with some everyday analogies:

1. The Problem: The "One-Size-Fits-All" Suit Didn't Fit

Previously, scientists tried to use "universal" robots trained on many languages (like Hindi or Urdu) to speak Kashmiri. They hoped the robot could just "guess" how to speak it.

• The Analogy: Imagine trying to wear a suit tailored for a giant to fit a child. It's too big, the sleeves are too long, and you can't move.
• The Reality: These universal robots failed miserably. They got a score of 1.86 out of 5 (where 5 is perfect). They sounded robotic and often unintelligible. Why? Because Kashmiri uses a special script (Perso-Arabic) with tiny marks called diacritics (like little hats or dots on letters). These marks change the sound of vowels completely. The old robots ignored these marks, leading to confusion.

2. The Solution: A Custom-Tailored Suit (Bolbosh)

The team built Bolbosh, a robot designed from the ground up to understand the specific "shape" of Kashmiri.

A. The "Acoustic Spa" (Cleaning the Data)

To teach the robot, they needed recordings of people speaking. They had two types of audio:

1. Studio recordings: Crystal clear, like a singer in a soundproof booth.
2. Spontaneous recordings: People talking in noisy markets or windy streets.

• The Analogy: If you try to teach a student by mixing a lecture from a quiet library with a shouting match at a construction site, the student gets confused.
• The Fix: They put the noisy recordings through a "3-stage spa treatment":
  1. Dereverberation: Removing the "echo" (like taking a photo out of a cave).
  2. Silence Trimming: Cutting out the awkward pauses.
  3. Loudness Normalization: Turning the volume up or down so everyone speaks at the same level.
     Now, the robot learns from a clean, consistent voice.

B. The "Language Translator" (Script-Awareness)

The robot needed to learn the alphabet.

• The Analogy: Imagine teaching someone to drive. If you give them a car with a steering wheel on the left, but the road signs are in a language they don't know, they will crash.
• The Fix: The team expanded the robot's vocabulary to include 272 specific characters, including all those tiny diacritic marks. They didn't just translate the letters; they taught the robot that a specific dot changes the meaning of a word entirely. This is called being "script-aware."

C. The "Smart Transfer" (Flow Matching)

This is the most technical part, but here is the simple version:

• The Analogy: Imagine you want to teach a chef how to cook a complex Kashmiri dish, but you only have 80 hours of data (which is very little for AI). Instead of starting from scratch (teaching them how to chop an onion), you hire a chef who is already an expert in Indian cooking (English/Indic languages).
• The Method: They took a robot that was already great at speaking English and "fine-tuned" it. They used a mathematical concept called Optimal Transport Flow Matching.
  • Think of this like a GPS navigation system. Instead of the robot guessing the route randomly, the GPS calculates the most efficient, smooth path from "Silence" to "Perfect Speech." This ensures the robot learns quickly and doesn't get stuck in a loop of bad sounds.

3. The Results: From Mumbling to Masterpiece

After all this work, they tested Bolbosh against the old "universal" robots.

• The Score:
  • Old Robots: 1.86/5 (Hard to understand).
  • Bolbosh: 3.63/5 (Very clear and natural).
• The Visual Proof: When they looked at the sound waves (spectrograms), the old robots looked like a blurry, smeared painting. Bolbosh looked like a sharp, high-definition photo with clear lines and distinct notes.

Why This Matters

This paper proves that for languages with complex scripts (like Kashmiri, Arabic, or Thai), you can't just use a "generic" AI. You have to respect the specific details of the script.

Bolbosh is a huge step forward. It means that in the future, a Kashmiri speaker can ask their phone to read a news article or a story, and the phone will actually sound like a human, not a glitchy robot. It bridges the gap between technology and culture, ensuring that 7 million people aren't left behind in the digital world.

Technical Summary

Here is a detailed technical summary of the paper "Bolbosh: Script-Aware Flow Matching for Kashmiri Text-to-Speech."

1. Problem Statement

Kashmiri, a Dardic language spoken by approximately 7 million people, is critically underserved in speech technology despite its official status and rich literary heritage. The development of Text-to-Speech (TTS) systems for Kashmiri faces three primary challenges:

• Data Scarcity: Paired text-speech corpora are fragmented and limited in quantity.
• Linguistic Complexity: Kashmiri is written in multiple scripts (primarily Perso-Arabic, Devanagari, and Roman). The Perso-Arabic script relies heavily on diacritics to encode subtle vowel distinctions essential for intelligibility.
• Failure of Zero-Shot Baselines: Existing multilingual TTS models trained on Indic languages (e.g., IndicParler) fail to produce intelligible Kashmiri speech. They achieve a low Mean Opinion Score (MOS) of 1.86 due to inadequate modeling of Perso-Arabic diacritics and phonotactic mismatches with high-resource training languages.

2. Methodology: The Bolbosh Framework

The authors propose Bolbosh, the first dedicated, open-source neural TTS system for Kashmiri. The framework utilizes a supervised cross-lingual adaptation strategy based on Optimal Transport Conditional Flow Matching (OT-CFM) within the Matcha-TTS architecture.

A. Core Architecture & Adaptation Strategy

• Backbone: The system is built on Matcha-TTS, which uses OT-CFM. Unlike diffusion models that require iterative denoising, OT-CFM learns a direct continuous-time vector field to transport a Gaussian prior to the target acoustic distribution. This ensures stable training dynamics and faster inference.
• Cross-Lingual Initialization: Instead of training from scratch, the model is initialized from a pretrained multi-speaker English Matcha-TTS checkpoint. This leverages transferable acoustic priors to accelerate convergence on low-resource data.
• Script-Aware Vocabulary Expansion: The model's vocabulary is expanded from standard English characters to 272 graphemes. This explicitly includes Kashmiri characters and diacritics, allowing the encoder to learn end-to-end grapheme-to-acoustic mappings without relying on external Grapheme-to-Phoneme (G2P) conversion (which is unavailable for Kashmiri).

B. Data Curation and Acoustic Enhancement

The training dataset consists of 79.9 hours of speech, combining:

1. RASA: Studio-quality, controlled recordings (primary backbone).
2. IndicVoices-R: Spontaneous, multi-environment recordings (provides speaker/prosodic diversity).

To unify these heterogeneous sources and stabilize alignment learning, a three-stage acoustic enhancement pipeline is applied to the spontaneous data:

1. Dereverberation & Denoising: Using the Resemble-Enhance framework (UNet-based denoiser + latent CFM refinement).
2. Dynamic Silence Trimming: Removing segments below 40 dB peak amplitude to prevent erroneous duration assignments.
3. Loudness Normalization: Standardizing to -23.0 LUFS followed by resampling to 22.05 kHz.

C. Training Objective

The model is fine-tuned using a multi-objective loss function:
$$L_{total} = L_{mel} + \lambda_{dur}L_{dur} + \lambda_{pitch}L_{pitch} + \lambda_{energy}L_{energy}$$

• $L_{mel}$: L1 reconstruction loss on mel-spectrograms.
• $L_{dur}$: Mean Squared Error (MSE) between predicted and Monotonic Alignment Search (MAS) derived durations.
• Prosody: Pitch and energy are modeled at the grapheme level.
• Regularization: Multi-speaker regularization is applied using learned speaker embeddings to prevent overfitting to the studio data, while inference conditions exclusively on RASA speaker embeddings for clarity.

3. Key Contributions

1. First Dedicated Kashmiri TTS: Introduction of the first script-aware, flow-matching-based TTS system specifically designed for Kashmiri.
2. Acoustic Domain Integration: Development of a principled enhancement pipeline that successfully unifies studio-quality and spontaneous speech corpora for low-resource adaptation.
3. Script-Aware Modeling: Demonstration that expanding the vocabulary to explicitly encode diacritics is critical for preserving fine-grained vowel distinctions and intelligibility.
4. New Benchmark: Establishment of a new state-of-the-art benchmark for Kashmiri speech synthesis, outperforming existing multilingual baselines significantly.

4. Results

The system was evaluated using both objective metrics and subjective human listening tests (32 native speakers).

Metric	Bolbosh (Ours)	IndicParler (Baseline)	Human (Ground Truth)
MOS (Mean Opinion Score)	3.63	1.86	4.61
MCD (Mel-Cepstral Distortion)	3.73	4.73	N/A
rWER (Relative Word Error Rate)	4.14%	46.75%	N/A

• Subjective Quality: Bolbosh achieved a MOS of 3.63, a massive improvement over the baseline's 1.86. The baseline frequently produced unintelligible or prosodically distorted speech, whereas Bolbosh achieved near-natural quality.
• Objective Fidelity: The MCD of 3.73 indicates significantly lower spectral distortion compared to the baseline (4.73).
• Diacritic Importance: When diacritics were removed from the input, Bolbosh's performance degraded (rWER rose to 13.23%), while the baseline collapsed completely (rWER 100.32%), proving that explicit diacritic modeling is essential for Kashmiri.
• Spectral Analysis: Spectrograms show that Bolbosh preserves clear harmonic structures and formant trajectories, whereas the baseline exhibits over-smoothing and temporal instability.

5. Significance

• Digital Inclusion: This work bridges the digital divide for 7 million Kashmiri speakers, enabling accessible voice-driven interfaces and human-computer interaction.
• Methodological Insight: The paper demonstrates that zero-shot multilingual transfer is insufficient for languages with complex diacritic systems. Instead, script-aware supervised adaptation combined with flow-based modeling is critical for low-resource TTS.
• Scalability: The proposed framework offers a scalable pathway for synthesizing high-quality speech in other severely under-resourced, diacritic-sensitive languages by leveraging pretrained acoustic priors and targeted vocabulary expansion.

The code and data for Bolbosh are publicly available at the provided GitHub repository, fostering further research in low-resource speech synthesis.