Technical Summary: The TTS–STT Flywheel for Indic ASR

1. Problem Statement

The paper identifies a critical gap in Indian-language Automatic Speech Recognition (ASR): while state-of-the-art (SOTA) open-source and commercial systems perform well on standard "read-prose" text (e.g., news, Wikipedia), they fail significantly on entity-dense audio. This niche domain includes real-world workflow content such as 10-digit phone numbers, currency amounts (in both Latin numerals and Indic words), Indian addresses with embedded Latin tokens, brand names, and English/Indic code-mix.

On a held-out, synthetic entity-dense Telugu test set, the open-source SOTA model (vasista22/whisper-telugu-large-v2) achieved an Entity-Hit-Rate (EHR) of only 0.027, and the commercial system (Deepgram Nova-3) achieved 0.16. Both systems perform orders of magnitude worse on this data compared to their read-prose performance, largely because their training corpora are dominated by read-prose text.

2. Methodology: The TTS–STT Flywheel

The authors propose a self-contained adaptation flywheel that uses open-source Text-to-Speech (TTS) to generate the specific training data required to close the ASR gap.

A. Entity-Dense Synthetic Audio (EDSA) Corpus

• Generation: The pipeline uses Anthropic Haiku-4.5 to generate ~22,000 entity-tagged carrier utterances across Telugu, Hindi, and Tamil. These are seeded with curated entity dictionaries (digits, currency, addresses, brands, proper nouns, code-mix).
• Synthesis Diversity: To prevent overfitting to a single voice distribution, utterances are routed across five TTS systems: Praxy R6 (open-source), Vanilla Chatterbox, IndicF5, ElevenLabs v3, and Cartesia sonic-3.
• Quality Control: A per-class Character Error Rate (CER) filter discards ~10–15% of synthetic clips that the baseline ASR cannot recognize.
• Text Alignment: A critical preprocessing step rewrites digit runs into their language-specific spelled-out forms (e.g., converting "54235" to "five lakh forty-two thousand...") to ensure the ground-truth text labels match the acoustic realization of the synthetic audio.
• Cost: The marginal cost for generating this corpus is under $50 (excluding free credits), a fraction of the cost of human transcription.

B. LoRA Fine-Tuning Recipe

The authors fine-tune ASR models using Low-Rank Adaptation (LoRA) on top of two base models:

1. Praxy-STT-rb (Headline): Fine-tunes vasista22 (Whisper-large-v2) on the EDSA corpus mixed with real-world data (IndicVoices, Common Voice, FLEURS). This model uses a smaller learning rate to avoid catastrophic forgetting of read-prose competence.
2. Praxy-STT-r2: Fine-tunes Whisper-large-v3 from scratch to demonstrate a language-conditional fix for "Script Collapse."

C. Evaluation Metrics

• Entity-Hit-Rate (EHR): A custom metric designed for semantic equivalence rather than token-level exact matches. It normalizes currency (handling "5 lakh" vs. "500,000"), brands, and spelled-out digits. It is deterministic and does not rely on LLM judges.
• Script Fidelity Rate (SFR): Measures the fraction of characters in the output that belong to the correct Unicode block for the target language, detecting "Script Collapse" (e.g., Telugu audio transcribed in Kannada script).

3. Key Results

A. Entity-Dense Recognition Performance

On the held-out synthetic entity-dense test set (using Cartesia audio not seen during training):

• Telugu: The flywheel model (Praxy-STT-rb) achieved an EHR of 0.473. This is a 17× improvement over the open-source SOTA (0.027) and a 3× improvement over the commercial Deepgram Nova-3 (0.16).
• Tamil: The model achieved an EHR of 0.543, a 22× improvement over both the open-source SOTA and Deepgram (both ~0.025).
• Hindi: The model achieved an EHR of 0.337 (7× over open-source SOTA), but underperformed Deepgram (0.485). The authors note this suggests the flywheel is most effective where commercial systems have not yet invested in entity coverage.

B. Native Speech Transfer

A sanity check using 20 native-human recorded Telugu utterances confirmed the model's ability to generalize beyond synthetic data. The EHR on native speech was 0.516, slightly higher than on the synthetic held-out set (0.473), indicating the gain is due to entity learning rather than TTS-distribution overfitting.

C. Read-Prose Regression

The authors report honest trade-offs. While entity recognition improved, there was a slight regression in read-prose Word Error Rate (WER):

• Telugu: +6.6 percentage points (pp) on FLEURS-Te.
• Hindi/Tamil: Regressions exceeded the pre-registered +7 pp threshold on FLEURS.
• However, on conversational holdouts (IndicVoices), the models maintained parity with the base models.

D. Language-Conditional Script Collapse

The paper identifies a specific failure mode in Whisper-large-v3 on Telugu, where it frequently outputs Kannada or Devanagari scripts (SFR 0.46–0.71). A per-language LoRA fix corrected this (SFR 0.81–0.97). However, the authors found this recipe contraindicated for Hindi and Tamil, where the vanilla model already has high script fidelity (SFR ≥0.98); applying the LoRA there caused significant WER regressions and SFR drops.

E. Ablation Study

An ablation training the same LoRA recipe on only read-prose data (FLEURS-Te) without the EDSA corpus resulted in an EHR of 0.020 (near the baseline). This conclusively attributes ~100% of the entity-recognition gain to the EDSA corpus, not the LoRA architecture itself.

4. Significance and Claims

The paper claims to close the gap in Indic ASR for niche, entity-dense domains where both open-source and commercial systems currently fail. Its significance lies in:

1. Cost-Effective Data Synthesis: Demonstrating that a high-quality, diverse training corpus for niche domains can be generated for <$50 using open-source TTS and LLMs, bypassing the high costs of human transcription.
2. Performance Leap: Achieving a 17–22× improvement in entity recognition on low-resource Indic languages, outperforming commercial APIs in Telugu and Tamil.
3. Methodological Rigor: Introducing the EHR metric to properly evaluate entity recognition and providing a diagnostic (SFR check) to prevent harmful fine-tuning on languages that do not need it.
4. Honest Reporting: The authors explicitly state that their models did not meet pre-registered targets (EHR ≥0.75 for Telugu) and that the task remains "substantially open," framing their results as a large step from a near-zero baseline rather than a solved problem.

All code, data, entity dictionaries, and model weights are released open-source to ensure reproducibility.