💬 NLP

Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages

This paper introduces a controlled, multidimensional pairwise evaluation framework for multilingual Text-to-Speech (TTS) systems in 10 Indian languages, utilizing over 120,000 human comparisons to construct a reliable leaderboard, analyze perceptual trade-offs, and interpret model preferences through Bradley-Terry modeling and SHAP analysis.

Original authors: Srija Anand, Ashwin Sankar, Ishvinder Sethi, Aaditya Pareek, Kartik Rajput, Gaurav Yadav, Nikhil Narasimhan, Adish Pandya, Deepon Halder, Mohammed Safi Ur Rahman Khan, Praveen S V, Shobhit Banga, Mite

Published 2026-04-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Srija Anand, Ashwin Sankar, Ishvinder Sethi, Aaditya Pareek, Kartik Rajput, Gaurav Yadav, Nikhil Narasimhan, Adish Pandya, Deepon Halder, Mohammed Safi Ur Rahman Khan, Praveen S V, Shobhit Banga, Mitesh M Khapra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine India as a giant, bustling marketplace where millions of people prefer to shout their orders, ask questions, and tell stories out loud rather than typing them on a keyboard. This is what the authors call a "Voice-First Nation."

But here's the problem: India is like a massive pot of spices. It has hundreds of languages, and people often mix them together in the same sentence (like saying "I need a chai and a bus ticket"). Making a computer speak all these languages clearly, naturally, and without sounding like a robot is incredibly hard.

This paper is essentially a massive "Taste Test" for computer voices.

Here is the breakdown of what they did, using simple analogies:

1. The Setup: The "Voice Olympics" 🏅

Instead of just asking people, "On a scale of 1 to 10, how good is this voice?" (which is like asking a judge to rate a gymnast's score out of 10), the researchers set up a Head-to-Head Tournament.

The Contestants: They picked 7 of the best "Text-to-Speech" (TTS) systems currently available. Some are famous commercial giants (like Google's Gemini or ElevenLabs), and some are open-source projects built specifically for Indian languages.
The Judges: They recruited over 1,900 native speakers from across India. These weren't just random people; they were trained to listen critically.
The Arena: They created a "gym" with 5,357 different sentences. These sentences were designed to be tricky:
- The "Tongue Twisters": Hard words to pronounce.
- The "Code-Mix": Sentences mixing English and Indian languages (e.g., "Please book a ticket for Mumbai").
- The "Stress Test": Sentences with numbers, math formulas, and acronyms.

2. The Game: The "Blind Taste Test" 🎧

The judges listened to two audio clips at the same time (Model A vs. Model B) without knowing which company made them.

They didn't just pick a winner. They acted like food critics rating a dish on six different ingredients:

Intelligibility: Can I understand the words?
Expressiveness: Does it sound happy, sad, or bored?
Voice Quality: Does it sound like a human or a robot?
Liveliness: Is it energetic or monotonous?
Hallucinations: Did the AI make up words that weren't in the text?
Noise: Is there static or hissing in the background?

3. The Results: Who Won? 🏆

After collecting over 120,000 comparisons, they used a mathematical formula (like a sports ranking system) to create a leaderboard.

The Champion: Gemini 2.5 Pro TTS took the gold medal. It was the most consistent winner across almost all languages and tricky sentence types.
The Runners-Up: ElevenLabs V3 and Sonic 3 were very close behind, often swapping places depending on the specific language.
The Underdog: The open-source model "Indic F5" came in last. While it tried to cover all 10 languages, it struggled to match the polish of the big commercial systems.

The Big Surprise: The researchers found that while "noise" and "hallucinations" (technical errors) are important, they aren't the main reason people pick a winner. Once the voice is clear and doesn't make mistakes, people choose the voice that sounds the most "alive" and "expressive." It's like choosing a singer: if they hit the right notes, you pick the one with the most soul.

4. The "How-To" Guide for Future Tests 📝

The paper also answers a practical question: "How many judges and how many sentences do we need to get a fair result?"

The Judges: You don't need thousands of people to get a stable ranking. About 200 good judges are enough to see who is clearly better than whom.
The Sentences: You need a diverse mix. About 1,000 different sentences covering different languages and topics are needed to make sure the ranking is fair.

The Takeaway

This paper is like a consumer report for AI voices. It tells us that:

We have a way to fairly test AI voices in complex, mixed-language environments.
Currently, the big commercial AI models are winning the race for Indian languages.
For the future, developers shouldn't just focus on making voices "error-free"; they need to focus on making them sound human, emotional, and lively, because that's what people actually care about.

The authors are releasing all their data and test sentences to the public, hoping to help other researchers build better, more human-like voices for the world's voice-first nations.

1. Problem Statement

India is a "voice-first" nation with immense linguistic diversity, including hundreds of languages and widespread code-mixing (mixing languages within a single sentence). While neural Text-to-Speech (TTS) systems have advanced, evaluating them in this context remains challenging due to:

Linguistic Diversity: The need to evaluate across 10+ Indic languages and complex phenomena like code-mixing, numerals, and domain-specific vocabulary.
Limitations of Traditional Metrics: Standard subjective metrics like Mean Opinion Score (MOS) rely on absolute ratings, which suffer from inter-rater calibration variance and lack diagnostic depth.
Scalability vs. Depth: Existing pairwise evaluations often focus only on overall preference, failing to explain why a system is preferred or how it performs across specific perceptual dimensions (e.g., expressiveness vs. noise).
Lack of Benchmarks: There is a scarcity of large-scale, controlled benchmarks for multilingual TTS in Indian languages that include code-mixed inputs and fine-grained perceptual analysis.

2. Methodology

The authors propose a controlled multidimensional pairwise evaluation framework designed to overcome the limitations of traditional MOS and scale to large datasets.

A. Benchmark Construction

Scale: 5,357 sentences across 10 Indic languages (Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Tamil, Telugu, Urdu).
Diversity: Covers 16 deployment-relevant domains (e.g., medical, legal, STEM, storytelling).
Input Types:
- Normalized: Text with numerals/acronyms verbalized.
- Symbolic: Raw text with formulas/operators.
- Code-mixed: Intra-sentential English insertions and transliteration.
- Edge Cases: Tongue twisters, extreme repetitions, and expressive utterances.
Quality Control: Sentences were curated from public sources or generated via LLMs, then strictly validated by native language experts for accuracy and fluency.

B. Data Collection

Participants: Over 1,900 native raters recruited across India.
Volume: Collected 120,000+ pairwise comparisons.
Protocol: A strict two-step workflow to prevent cognitive overload:
1. Holistic Preference: Raters listen to two anonymous models (A vs. B) and select a winner (or "Both Good/Bad").
2. Granular Diagnosis: Raters then evaluate the same pair across 6 perceptual axes: Intelligibility, Expressiveness, Voice Quality, Liveliness, Noise, and Hallucinations.

C. Statistical Modeling

Ranking: Used Bradley-Terry (BT) modeling to convert pairwise preferences into a latent score leaderboard.
Uncertainty: Applied bootstrap resampling (500 iterations) to generate 95% confidence intervals. A system is considered strictly better only if its confidence interval lies entirely above another's.
Analysis: Used SHAP (SHapley Additive exPlanations) analysis on an XGBoost classifier to determine which perceptual axes drive overall preference.

3. Key Contributions

Large-Scale Benchmark: The creation of a phonetically diverse, multilingual TTS benchmark covering 10 languages and complex real-world inputs (code-mixing, numerals).
Multidimensional Framework: Moving beyond "overall preference" to a 6-axis diagnostic framework that isolates specific failure modes (e.g., hallucinations vs. lack of expressiveness).
Statistical Rigor: A methodology that quantifies leaderboard reliability, determining the minimum number of raters and sentences required for stable rankings.
Insight into Human Preference: Demonstrating that while robustness (low noise/hallucinations) is a baseline requirement, expressiveness and intelligibility are the primary drivers of human preference in high-quality systems.

4. Key Results

A. System Performance (Leaderboard)

Evaluated 7 state-of-the-art TTS systems (including commercial APIs like GEMINI 2.5 PRO, ELEVEN LABS, and open-source models like INDIC F5).

Top Performer: GEMINI 2.5 PRO TTS ranked first with a BT score of 1128.53, consistently leading across 9/10 languages and all 16 domains.
Runners-up: ELEVEN LABS V3 and SONIC 3 followed closely, showing statistical parity in some languages.
Open Source Gap: INDIC F5 (an Indic-specialized model) ranked last (805.75), indicating a significant performance gap compared to commercial giants despite covering all 10 languages.
Robustness: GEMINI 2.5 PRO maintained top rankings across all input types (Normalized, Symbolic, Code-mixed), demonstrating superior handling of edge cases.

B. Perceptual Drivers (SHAP Analysis)

Primary Drivers: Expressiveness and Intelligibility were the strongest predictors of overall preference.
Secondary Drivers: Liveliness and Voice Quality.
Baseline Factors: Noise and Hallucinations contributed less to differentiation because most evaluated systems had already achieved a high baseline of robustness; thus, they were "table stakes" rather than differentiators.

C. Reliability and Sample Efficiency

Rater Count: Rankings stabilize (Spearman's $\rho \ge 0.95$ ) with approximately 100–200 raters, depending on the number of systems compared.
Sentence Count: Approximately 1,000 sentences are required for stable rankings in 5–7 system comparisons. Increasing sentences beyond this reduces score uncertainty but yields diminishing returns on rank stability.

5. Significance

Standardization: Provides a reproducible, statistically grounded framework for evaluating multilingual TTS, addressing the "black box" nature of subjective listening tests.
Diagnostic Value: The 6-axis breakdown allows developers to pinpoint specific weaknesses (e.g., a model might be intelligible but lack liveliness), guiding targeted improvements.
Voice-First Accessibility: By focusing on Indian languages and code-mixing, the study directly supports the development of accessible digital services for a massive, linguistically diverse population.
Resource Release: The authors commit to releasing the benchmark and preference data, fostering future research in multilingual speech synthesis.

In conclusion, this paper establishes that for multilingual TTS in India, quality is multidimensional. While robustness is necessary, the systems that win user preference are those that master the nuances of expressiveness and intelligibility across complex, code-mixed inputs. The study also provides practical guidelines for researchers on how to design efficient, reliable human evaluation studies.