💻 computer science

IndicContextEval: A Benchmark for Evaluating Context Utilisation in Audio Large Language Models Across 8 Indic Languages

This paper introduces IndicContextEval, a comprehensive multilingual benchmark spanning 8 Indian languages and 23 domains, designed to rigorously evaluate whether Audio Large Language Models genuinely utilize explicit contextual prompts or rely on pretraining knowledge through a novel 7-level prompting framework.

Original authors: Sakshi Joshi, Dhruv Subhash Rathi, Sanskar Singh, Eldho Ittan George, R J Hari, Kaushal Bhogale, Mitesh M. Khapra

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sakshi Joshi, Dhruv Subhash Rathi, Sanskar Singh, Eldho Ittan George, R J Hari, Kaushal Bhogale, Mitesh M. Khapra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual assistant who can listen to people speaking and write down exactly what they say. This is what AudioLLMs (Audio Large Language Models) do. They are like super-powered dictation tools that can also listen to a little note you give them beforehand, like "This is a medical meeting" or "Here is a list of names to watch for."

The big question the authors of this paper asked is: Does this assistant actually read your note and use it to help, or does it just pretend to listen while relying on what it already memorized during its training?

To find the answer, they built a new testing ground called IndicContextEval. Here is a simple breakdown of what they did and what they found, using some everyday analogies.

1. The Test Kitchen: IndicContextEval

Think of this benchmark as a massive, organized kitchen where they cooked up 56 hours of natural speech from 555 different people speaking 8 different Indian languages (like Hindi, Bengali, and Tamil).

The Ingredients: They didn't just record random chatter. They focused on 23 different professional fields, ranging from "Robotics and Automation" to "Culinary Arts" and "Law." This ensures the speakers use tricky, technical words that are hard to guess just by hearing them.
The Goal: They wanted to see if the AI could correctly transcribe these tricky words when given a "hint" (context) versus when it had to guess on its own.

2. The Seven Levels of Hints (The "Prompt" Ladder)

The researchers designed a clever ladder of 7 levels to test the AI. Imagine you are asking a friend to write down a story about a specific topic. You give them hints that get more and more specific:

Level 0 (The Blank Slate): "Just write down what you hear." (No hints at all).
Level 1 (The Language Hint): "Write down what you hear in Hindi." (Just tells the language).
Level 2 (The Structured Note): "This is a medical talk in Hindi, spoken by a doctor." (Facts only).
Level 3 (The Story Note): "This is a doctor explaining a surgery in Hindi." (A natural sentence description).
Level 4 (The English List): "Here is a list of medical terms in English: Heart, Scalpel, Antibiotic." (But the AI must still write the answer in Hindi).
Level 5 (The Native List): "Here is the same list, but written in Hindi script." (The hint matches the answer language).
Level 6 (The Trap): "Here is a list of cooking terms (like Flour, Oven, Spice) for a medical talk." (This is a trick to see if the AI blindly follows the list even when it's wrong).

3. The Results: Who Actually Listens?

They tested five different AI models. Here is what happened, using the "Friend" analogy:

The "Smart & Skeptical" Friend (GPT-4o Transcribe):
This friend is good. When you give them the right list of words (Level 5), they do a great job. But when you give them the wrong list (Level 6, the cooking trap), they ignore it and stick to what they actually hear. They know when to use the hint and when to ignore it.
The "Eager Learner" (Gemini 3 Flash):
This friend loves the hints. When you give them the right list, they improve their writing significantly. They seem to genuinely use the context to get the tricky words right.
The "Blind Follower" (Gemma-3N):
This friend is too trusting. When you give them the wrong list (Level 6), they get confused and start writing nonsense based on that wrong list. They rely too much on the note and forget to listen to the actual voice.
The "Deaf to Notes" Friend (Sarvam Audio):
This friend is actually the best at just listening and writing down words (lowest errors overall). However, they barely change their behavior whether you give them a hint or not. It's like they are so good at hearing that they don't really need the notes, or they simply ignore them.
The "Confused" Friend (GPT-4o and others at Level 0):
When no language was specified (Level 0), many models got confused about which language script to use, making lots of mistakes. Once you told them "Speak Hindi," their performance jumped up immediately.

4. Key Takeaways

The "Hint" Format Matters: Telling the AI "Here is a list of words in English" (Level 4) wasn't as helpful as giving the list in the local language script (Level 5). It's like trying to read a recipe written in a language you don't speak; it doesn't help as much as having it in your own language.
Natural vs. Robot: Describing the audio in a natural sentence (Level 3) worked better than giving a dry list of facts (Level 2).
The Trap Test: The "Level 6" trap was crucial. It proved that some models are smart enough to ignore bad hints, while others are so eager to please that they follow bad instructions and make mistakes.

Summary

The paper concludes that while these AI models are powerful, they don't all use context in the same way. Some are smart enough to know when to use a hint and when to ignore it. Others are either too blind to the hints or too easily tricked by them.

The authors built this test (IndicContextEval) to help developers understand these differences so they can build better, more reliable voice assistants for Indian languages in the future. They made all their data and tests public so others can continue this research.

Technical Summary: IndicContextEval

Problem Statement

Audio Large Language Models (AudioLLMs) have introduced the capability to perform speech recognition conditioned on free-form textual prompts, such as domain descriptions or entity lists. While this offers a flexible interface for contextual Automatic Speech Recognition (ASR), a critical uncertainty remains: do these models genuinely utilize the provided context to resolve acoustic ambiguities, or do they rely primarily on parametric knowledge acquired during pre-training?

Existing benchmarks fail to address this question. Large multilingual corpora (e.g., IndicVoices, CommonVoice) evaluate transcription under fixed prompting conditions without varying contextual inputs. Conversely, benchmarks designed for contextual ASR (e.g., ProfASR, ContextASR) are predominantly English-centric, often rely on synthetic speech, and typically test only a single mode of context (e.g., named entity lists). Consequently, there is no systematic method to distinguish between genuine contextual grounding and parametric memorization, nor to analyze how specific types of context influence transcription behavior.

Methodology: The IndicContextEval Benchmark

To address these gaps, the authors introduce IndicContextEval, a multilingual benchmark designed for the controlled evaluation of context utilization in AudioLLMs.

Dataset Construction

Scale and Diversity: The benchmark comprises 55.93 hours of natural speech from 555 speakers across 8 Indian languages (Hindi, Bengali, Telugu, Marathi, Gujarati, Malayalam, Odia, and Urdu) and 23 professional domains (ranging from Core Engineering and Medical Sciences to Forensics and Arts).
Recording Styles: Data includes both read speech (using curated domain vocabulary lists translated from English) and extempore speech (participants discussing technical concepts in a natural narrative style).
Quality Control: All recordings underwent multi-stage verification by native speakers and domain experts. Reference transcriptions were produced verbatim in native scripts, preserving code-mixed segments, with English entities transliterated and provided in brackets.

Controlled Prompt Taxonomy (L0–L6)

The core innovation is a seven-level prompting framework where each level introduces exactly one additional contextual signal while holding other factors constant. This design allows for the attribution of performance changes to specific context types:

L0: No context (bare transcription instruction).
L1: Language only (target language specified).
L2: Language + Structured domain metadata (speech style, region, domain description).
L3: Language + Natural-language audio description (topic and discussion type).
L4: Language + Entity list in English (20–30 domain entities).
L5: Language + Entity list in native script (aligning prompt and output scripts).
L6: Adversarial prompt (Entity list from an unrelated domain in native script).

Experimental Setup

The study evaluated five models:

IndicConformer: A standalone ASR baseline (evaluated at L1 only).
GPT-4o Transcribe, Gemini 3 Flash, Sarvam Audio: Commercial AudioLLMs.
Gemma-3N: An open-weight AudioLLM.

Evaluation metrics included Word Error Rate (WER) and Named Entity Error Rate (NEER), with a specific focus on how NEER changes across context levels to measure entity biasing.

Key Results

1. Baseline Performance and Language Identification

At L1 (language specified), Sarvam Audio achieved the lowest WER (16.86%), followed by Gemini 3 Flash (18.90%). GPT-4o Transcribe and Gemma-3N exhibited significantly higher error rates. The transition from L0 to L1 revealed a substantial "language identification tax," particularly for Gemma-3N, which improved by 12.48 WER points when the language was explicitly specified, indicating difficulty in inferring the correct script from acoustics alone.

2. Impact of Context Type

Natural Language vs. Metadata: Natural-language audio descriptions (L3) consistently outperformed structured metadata prompts (L2). For instance, GPT-4o Transcribe improved by 2.53 WER points at L3 versus only 0.24 at L2. Gemma-3N showed extreme sensitivity to prompt format, with structured metadata degrading performance significantly (+13.47 WER).
Script Alignment: Providing entity lists in the native script (L5) yielded the largest gains in NEER across all models. The gap between English-script entities (L4) and native-script entities (L5) reached up to 11 WER points, confirming a substantial "script-mismatch cost."
Adversarial Robustness (L6):
- GPT-4o Transcribe demonstrated balanced utilization: it benefited from correct entities (L5) but remained robust to incorrect entities (L6 $\approx$ L1), suggesting it cross-validates prompts against acoustic evidence.
- Gemini 3 Flash showed moderate sensitivity to adversarial prompts, indicating genuine contextual use.
- Gemma-3N was heavily degraded by adversarial entities (+9.22 WER), indicating a "blind reliance" on entity prompts that often corrupted the surrounding transcript.
- Sarvam Audio remained largely unaffected by adversarial prompts, suggesting its transcription is driven primarily by the acoustic encoder with limited influence from textual context.

3. Contextual Grounding Capabilities

The results highlight substantial heterogeneity in how models utilize context. While native-script entity biasing provided the strongest improvements, some models (like Sarvam Audio) showed minimal variation across context levels, implying they largely ignore textual prompts. Conversely, others (like Gemma-3N) relied on prompts to the point of instability. Notably, at L5, even the best-performing AudioLLM (Gemini 3 Flash) surpassed the standalone ASR baseline (IndicConformer) in both WER and NEER.

Significance and Claims

The paper claims that IndicContextEval is the first benchmark to systematically vary context types while holding other factors constant across multiple Indic languages and professional domains. Its primary significance lies in:

Revealing Divergent Behaviors: It exposes that AudioLLMs vary widely in their ability to ground transcription in context, with some models effectively exploiting signals, others ignoring them, and some responding unstably.
Distinguishing Mechanisms: The controlled taxonomy allows researchers to distinguish between genuine contextual grounding and parametric memorization, a distinction existing benchmarks cannot make.
Highlighting Open Challenges: Despite flexible prompting interfaces, the paper concludes that contextual grounding remains an open challenge for AudioLLMs. The findings suggest that current models do not uniformly or reliably integrate contextual signals, necessitating explicit evaluation of these capabilities before deployment in critical applications.

The authors emphasize that these findings are derived from systematic experimentation and that the benchmark resources are publicly available to support future research into robust, context-aware speech recognition.