⚡ electrical engineering

Praxy Voice: Voice-Prompt Recovery + BUPS for Commercial-Class Indic TTS from a Frozen Non-Indic Base at Zero Commercial-Training-Data Cost

Praxy Voice achieves commercial-class Text-to-Speech performance for Telugu, Tamil, and Hindi by combining a Brahmic Unified Phoneme Space (BUPS) for script conversion, a text-only LoRA adapter trained on licensed data, and a specific voice-prompt sampling recipe, all without requiring commercial training data or acoustic decoder retraining.

Original authors: Venkata Pushpak Teja Menta

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Venkata Pushpak Teja Menta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a world-class chef (the Chatterbox AI) who can cook delicious meals for 23 different languages, like English, Spanish, and French. However, this chef has never cooked for three specific regions: Telugu, Tamil, and Hindi. If you ask them to cook a Telugu dish, they either refuse or serve you a strange, inedible mix of ingredients because they don't know the local recipes.

The paper introduces Praxy Voice, a clever "kitchen hack" that allows this same chef to cook perfect Telugu, Tamil, and Hindi meals without hiring a new chef, without buying new ingredients (commercial training data), and without teaching the chef how to cook from scratch.

Here is how they did it, broken down into three simple steps:

1. The Universal Translator (BUPS)

The Problem: The chef only understands Latin letters (A, B, C). If you write a recipe in Telugu script (which looks like circles and lines), the chef sees it as gibberish.
The Fix: The team built a tool called BUPS. Think of it as a magic translator that instantly converts the Telugu, Tamil, and Hindi scripts into Latin letters before the chef sees them.

Analogy: It's like taking a handwritten letter in a foreign script, scanning it, and having a computer instantly type it out in English so the chef can read it. The meaning and sounds stay the same, but now the chef can actually process the instructions.

2. The Specialized Sous-Chef (LoRA Adapter)

The Problem: Even though the chef can now read the recipe (thanks to BUPS), they still don't know how to cook it. They might read the word "curry" but cook it like a stew because they've never made Indian food before.
The Fix: The team added a tiny, specialized assistant (a LoRA adapter) to the part of the chef's brain that reads recipes.

The Magic: They trained this assistant on about 1,220 hours of licensed Indian audio (like public domain podcasts and news).
The Trick: Since the chef doesn't know Telugu or Tamil, the team told the assistant to pretend the recipes were Hindi (which the chef does know). Because Telugu and Hindi sound somewhat similar, the chef uses their existing "Hindi cooking style" as a base and the assistant tweaks it just enough to make it sound like Telugu or Tamil.
Result: This assistant is tiny (less than 1% of the chef's total brain power) but makes a huge difference.

3. The "Taste Test" Recipe (Voice Prompt & Config B)

The Problem: Even with the translator and the assistant, the chef's voice sounds a bit "foreign" or robotic when speaking these new languages. It's intelligible, but it lacks the natural rhythm and accent of a native speaker.
The Fix: The team discovered a specific "recipe" for the final step:

The Voice Prompt: Before cooking, you play the chef an 8–11 second clip of a native speaker's voice. This acts like a "taste test," telling the chef, "This is exactly how the dish should sound."
The "Config B" Settings: They tweaked three dials on the chef's oven (temperature, exaggeration, and filtering).
- Analogy: Imagine the chef was cooking at a low simmer. The team turned up the heat slightly and adjusted the seasoning to make the flavors pop. These specific settings (exaggeration 0.7, temperature 0.6, min_p 0.1) were found to be the "sweet spot" for making the voice sound natural.

The "Two-Branch" Strategy

The team noticed something interesting: If they used this special assistant on Hindi (which the chef already knows how to cook), the food actually got worse. The assistant confused the chef.

The Solution: They built a smart switch.
- If the input is Telugu or Tamil: Use the Translator + Assistant + Voice Prompt.
- If the input is Hindi: Skip the assistant and just use the original chef + Voice Prompt.
- This ensures the best quality for every language.

Handling the "Mixed Language" Problem

In India, people often mix English words into Indian sentences (e.g., "I sent a WhatsApp message").

The Problem: The main chef (Chatterbox) gets confused by the English words mixed in and drops them silently.
The Fix: For these mixed sentences, they use a different, pre-trained chef (IndicF5) that is better at handling mixed languages. However, this chef also struggles with English words written in Latin script.
The Hack: They use a small AI tool to rewrite the English words into Indian script phonetically (e.g., changing "WhatsApp" to "व्हाट्सऐप") before feeding it to the chef. This stops the chef from dropping the words.

The Results

When they tested this system against the best commercial voice systems (like ElevenLabs or Cartesia):

Telugu: It made fewer pronunciation errors than the commercial leaders.
Tamil: It handled a specific difficult sound ("zha") much better than the competition.
Hindi: It matched the best commercial systems in understanding and clarity.

The Bottom Line:
The authors proved that you don't need millions of dollars or thousands of hours of training to get a "commercial-grade" voice for Indian languages. By using a smart translator, a tiny specialized assistant, and a specific "voice prompt" recipe, they turned a general-purpose AI into a native-sounding Indian voice actor for free (using only open-source tools and public data).

They have released all the code and the "assistant" weights for anyone to use.

1. Problem Statement

Building high-quality Text-to-Speech (TTS) systems for Indic languages (specifically Telugu, Tamil, and Hindi) typically requires either:

Training from scratch: Requiring hundreds to thousands of GPU-hours and massive datasets (e.g., Indic Parler-TTS, IndicF5).
Using closed commercial APIs: Which incur per-call costs and lack data control.

Existing open-source multilingual bases, such as ResembleAI Chatterbox, support 23 languages (including Hindi) but fail to natively support Telugu and Tamil. Furthermore, they lack tokenization capabilities for Brahmic scripts (Devanagari, Telugu, Tamil, etc.), often resulting in tokenization errors or "byte-BPE junk" when processing these languages. The authors ask: What is the minimum intervention required to bring a frozen, non-Indic-native multilingual base to commercial-class performance on Telugu, Tamil, and Hindi without training a new acoustic decoder or using commercial TTS training data?

2. Methodology

The authors propose a three-branch inference pipeline that combines three engineering components into a single recipe. The core innovation is adapting a frozen model via minimal parameter updates and inference-time heuristics.

A. BUPS (Brahmic Unified Phoneme Space)

Challenge: Chatterbox's tokenizer cannot process raw Telugu or Tamil scripts.
Solution: A deterministic routing layer that converts Brahmic scripts (Devanagari, Telugu, Tamil, Kannada, Bengali, Gujarati, Malayalam) into ISO-15919 romanized Latin strings at tokenization time.
Mechanism: It segments input text by script, transliterates Brahmic spans to Latin using diacritics (preserving phonology), and passes non-Brahmic spans unchanged. This allows the model's existing Latin-tokenizer paths to process Indic text without modifying the model weights.

B. LoRA Adaptation (Text Head Only)

Target: Only the text-token predictor (t3 transformer) of the Chatterbox base. The acoustic decoder (s3gen) and voice encoder (ve) remain frozen.
Configuration: A Low-Rank Adaptation (LoRA) adapter applied to attention projections ( $q, k, v, o$ $q, k, v, o$ ).
- Trainable Parameters: 7.86M (0.97% of the 810M base model).
- Training Data: ~1,220 hours of licensed Indic audio (IndicTTS, Rasa, FLEURS, Shrutilipi). Zero commercial TTS data was used.
- Language Conditioning: Telugu and Tamil texts are conditioned with a Hindi-proxy language ID (hi). This routes the adaptation through the closest supported acoustic manifold the base already possesses.
Result: The text head learns to emit speech tokens that align with Indic phonology, while the frozen decoder generates the audio.

C. Voice-Prompt Recovery & Config B

Problem: Even with BUPS and LoRA, default inference yields "flat" or "foreign-accented" speech because the frozen decoder lacks native Indic acoustic priors.
Solution: An inference-time recipe involving:
1. Voice Prompt: Supplying an 8–11 second reference audio clip in the target language (same language as the input text). This conditions the decoder on native prosody and timbre.
2. Config B Sampling Overrides: Three specific parameter adjustments to stabilize generation:
  - exaggeration: 0.7 (increases prosodic contour).
  - temperature: 0.6 (tightens distribution).
  - min_p: 0.1 (filters low-probability tokens causing phoneme drift).

D. Routing Architecture

The system deploys a two-branch architecture for pure-script inputs and a third branch for code-mixed inputs:

Telugu/Tamil (Pure Script): Routes through BUPS + LoRA (Hi-proxy).
Hindi (Pure Script): Routes through Vanilla Chatterbox (No LoRA, No BUPS). Crucially, applying LoRA to Hindi regresses semantic accuracy, so a negative control branch is used.
Code-Mixed (e.g., Hindi + English): Routes to IndicF5 (a native Indic flow-matching model) preceded by a Haiku-driven transliteration preprocessor that converts English words into native-script phonetic spellings (e.g., "WhatsApp" $\to$ "व्हाट्सऐप").

3. Key Contributions

BUPS: The first application of ISO-15919 romanization as a routing mechanism to enable Indic support in Latin-tokenized TTS bases.
Minimum-Intervention LoRA: A strategy adapting only the text head of a frozen multilingual model using a proxy language ID, achieving commercial quality without acoustic decoder training.
Voice-Prompt Recovery Recipe: Demonstrates that specific sampling overrides (Config B) combined with same-language voice prompts can recover commercial-class acoustics from a frozen decoder.
Two-Branch Deployment: A novel architecture that dynamically switches between LoRA-adapted and vanilla models based on language, validated by a negative control showing LoRA harms Hindi performance.
Code-Mix Solution: A pre-processing pipeline that reduces LLM-WER on code-mixed inputs from ~0.80 to ~0.14–0.27 by transliterating English spans to native scripts.
Open Release: R6 LoRA weights (Apache-2.0) and inference code (MIT) are publicly available.

4. Results

Evaluated on the PSP (Phoneme Substitution Profile) benchmark and intelligibility metrics:

Telugu:
- Retroflex Collapse: 26.7% (Praxy) vs. 33.3% (Sarvam Bulbul) and 50% (Cartesia). Praxy ranks first.
- FAD (Fréchet Audio Distance): 291.3 (comparable to commercial baselines).
Tamil:
- Zha Collapse: 71% (Praxy) vs. 86% (Commercial Trio). This is the clearest per-dimension gain observed.
- PSD (Prosodic Signature Divergence): 71.2 (matches Sarvam Bulbul).
Hindi:
- LLM-WER: 0.025 (Praxy Vanilla) ties with Cartesia Sonic-3.
- Note: The LoRA branch significantly degraded Hindi performance (LLM-WER 0.334), validating the need for the two-branch routing.
Code-Mixed:
- LLM-WER dropped from 0.80–0.85 (raw IndicF5) to 0.14–0.27 with the transliteration preprocessor.

5. Significance

Cost Efficiency: Achieves commercial-grade TTS for major Indic languages using zero commercial training data and minimal compute (single A100, ~11 hours, ~$45).
Accessibility: Democratizes access to high-quality Indic TTS for developers who cannot afford the compute or data required to train models from scratch.
Methodological Insight: Proves that for multilingual bases, text-head adaptation combined with inference-time acoustic conditioning (voice prompts) is sufficient to bridge the gap for unsupported languages, provided the base has a "close enough" acoustic manifold (e.g., Hindi for Telugu/Tamil).
Negative Control: The paper rigorously demonstrates that "more adaptation" is not always better; the LoRA adapter actively harms performance on languages the base already supports (Hindi), necessitating a conditional routing strategy.

The authors conclude that this "minimum-intervention" approach effectively closes the gap between open-source and commercial TTS for pure-script Indic inputs, with a separate, low-cost solution for code-mixed scenarios.