Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages

Technical Summary: Indic DiarBench

Problem Statement
While significant progress has been made in Automatic Speech Recognition (ASR) for Indian languages, existing research and benchmarks predominantly focus on single-speaker scenarios. Real-world applications such as meetings, interviews, and casual conversations involve multiple interacting speakers, necessitating a pipeline that combines speaker diarization (identifying "who spoke when") with ASR. Current benchmarks for Indian languages are insufficient because they either lack speaker-attributed labels, cover only a subset of languages, or decouple diarization evaluation from ASR evaluation. This decoupling masks critical failure modes; for instance, ASR performance often degrades significantly when applied to the short, fragmented, and overlapping segments produced by diarization systems. There is a lack of a standardized, multilingual benchmark that evaluates joint speaker-attributed ASR across India's 22 scheduled languages under realistic conversational conditions.

Methodology
To address this gap, the authors introduce Indic DiarBench, a comprehensive benchmark dataset and evaluation framework.

• Data Collection: The corpus comprises approximately 108 hours of natural multi-speaker audio spanning all 22 scheduled Indian languages. The data is categorized into three acoustic conditions:

  1. Near-field meetings (~53 hrs): Recorded via individual close-proximity microphones in virtual meetings, capturing spontaneous interaction and overlap.
  2. Far-field meetings (~27 hrs): Recorded with distant microphones, introducing reverberation and background noise.
  3. In-the-wild audio (~28 hrs): Curated from YouTube videos featuring sustained multi-speaker interactions in unconstrained environments.
     The dataset includes 485 unique speakers from 189 districts, ensuring diverse geographic, demographic, and dialectal representation.

• Annotation Pipeline: A rigorous human-in-the-loop process was employed to generate high-quality, speaker-attributed transcriptions:

  1. Bootstrap: Initial transcripts were generated using multiple ASR systems to reduce annotation effort.
  2. Human Verification: Professional annotators corrected word sequences, timestamps, and speaker labels. For meeting recordings, annotators adjusted existing labels but could not introduce new speakers; for in-the-wild data, they could merge or add speakers based on visual cues.
  3. Code-Mixing Handling: Two transcription formats were created: one in native Indic scripts (including English words) and one normalized form (English in Roman script, numerals in Arabic digits) to accommodate frequent code-mixing.
  4. Quality Control: Dedicated checkers and language-specific experts reviewed consistency, overlap handling, and non-speech tags (, ).

• Evaluation Metrics: The benchmark evaluates systems on two axes:

  • Acoustic Diarization: Measured via Diarization Error Rate (DER) without a forgiveness collar, including overlapping speech.
  • Joint Performance: Measured via Concatenated minimum-permutation Word Error Rate (cpWER) to assess transcription accuracy after resolving speaker permutations, and Word Diarization Error Rate (WDER) to explicitly penalize speaker attribution errors.

Key Contributions

1. First Multilingual Benchmark: The first open-access resource covering all 22 scheduled Indian languages with joint ASR and diarization labels.
2. Diverse Acoustic Conditions: A unified corpus of ~108 hours covering near-field, far-field, and in-the-wild scenarios, capturing realistic variations in speaker counts (2–9 speakers) and overlap ratios.
3. Human-Corrected Ground Truth: High-quality, time-aligned speaker-attributed transcriptions (RTTM format) with rigorous quality control, including specific handling for code-mixing and overlapping speech.
4. Baseline Evaluations: Comprehensive benchmarking of leading systems, including commercial speech APIs (e.g., AWS, Azure, Deepgram), multimodal Large Language Models (e.g., GPT-4o, Gemini 3 Pro), and Indic-specialized systems (Sarvam AI).

Results and Analysis
The authors evaluated multiple state-of-the-art systems on the dataset:

• Performance Leaders: The Indic-specialized Sarvam AI pipeline achieved the best overall results, with the lowest DER (16.0%) and cpWER (38.8%).
• Commercial APIs: Systems like AWS Transcribe showed moderate performance (23.5% DER), while others exhibited substantially higher error rates.
• Multimodal LLMs: These models presented a trade-off. Gemini 3 Pro achieved competitive WDER (33.0%) due to strong ASR on detected segments but suffered from very high DER (74.0%) due to missed detections and unreliable timestamping. GPT-4o showed better diarization accuracy but poor cpWER, particularly on lower-resource languages.
• Error Patterns: Analysis revealed that Indic-specialized pipelines had balanced error profiles, whereas multimodal LLMs were dominated by missed detection errors (e.g., failing to detect short utterances like affirmations). Generic commercial APIs often exhibited high missed detection rates due to conservative voice-activity detection.
• Language and Condition Variance: Performance varied significantly by language and condition. Telugu, Maithili, and Dogri were the most challenging due to high overlap ratios and speaker counts. Santali and Urdu were among the easiest. In-the-wild recordings generally yielded the best performance due to lower overlap and cleaner turn-taking. Lower-resource languages (available only in near-field) showed higher error rates compared to higher-resource languages spanning multiple conditions.

Significance
Indic DiarBench establishes a standardized evaluation framework for multi-speaker conversational speech in Indian languages. By providing a resource that captures the nuances of Indian speech—such as code-mixing, dialectal variation, and frequent speaker overlap—the benchmark enables the development of robust, speaker-attributed ASR systems. The authors posit that progress in real-world conversational transcription requires benchmarks that evaluate ASR and diarization jointly, as independent evaluation fails to capture the degradation caused by diarization errors. The release of this dataset, along with annotations and baseline systems, aims to advance inclusive, multilingual speech technology research for the 22 constitutionally recognized languages of India.