💬 NLP

IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages

This paper introduces IndicIFEval, a benchmark comprising 800 human-verified examples across 14 Indic languages to evaluate LLMs' instruction-following capabilities, revealing that while models adhere well to formatting constraints, they significantly lag behind English performance in lexical and cross-lingual tasks.

Original authors: Thanmay Jayakumar, Mohammed Safi Ur Rahman Khan, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan

Published 2026-02-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thanmay Jayakumar, Mohammed Safi Ur Rahman Khan, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, multilingual robot assistant. You can ask it to write a story, summarize a news article, or solve a math problem. But there's a catch: you also want to give it very specific rules, like "Write exactly three sentences," "Don't use the word 'cat'," or "Start your answer with the word 'Hello'."

This paper, IndicIFEval, is like a giant, rigorous "driver's license test" for these robot assistants, but specifically for 14 different Indian languages (like Hindi, Bengali, Tamil, and Sanskrit).

Here is the breakdown of what they did, using some simple analogies:

1. The Problem: The "English-Only" Bias

Imagine a driving school that only tests you on English roads. You might be a perfect driver in English, but if you move to a country where they drive on the left, speak a different language, and have different traffic signs, you might crash.

Currently, most AI tests are built in English. The researchers found that while these AI models are great at following rules in English, they often get confused, forget instructions, or make up their own rules when asked to do the same thing in Indian languages. There was no good way to measure how bad this was, so they built a new test.

2. The Solution: Two Types of Tests

To make the test fair and thorough, they created two different "exam rooms" (datasets):

Room A: The "Translation" Exam (INDICIFEVAL-TRANS)
- The Analogy: Imagine taking a famous English exam and translating it word-for-word into Hindi or Tamil.
- The Goal: This checks if the AI can handle the structure of the instruction. However, the researchers were careful. They didn't just use Google Translate. They had human experts tweak the questions so they made sense culturally. For example, changing "The President of the USA" to "The Prime Minister of India" so the AI doesn't get confused by the context.
- The Catch: Sometimes, a direct translation feels "stiff" or unnatural, like wearing a suit that was tailored for someone else's body.
Room B: The "Native" Exam (INDICIFEVAL-GROUND)
- The Analogy: Instead of translating an English exam, imagine writing brand-new questions from scratch, based on local culture, news, and stories that naturally happen in India.
- The Goal: This tests if the AI can follow rules when the instructions feel like they were written by a local person, not a translator. It's like asking the AI to write a poem about a local festival with specific rhyming rules, rather than translating a poem about Christmas.

3. The Test Subjects: The AI Models

They put over 20 different AI models through these tests. Some were "Open-Weight" (like open-source software anyone can download and tweak) and some were "Proprietary" (like the secret, high-end models from big tech companies).

They tested models of all sizes, from tiny ones (like a pocket calculator) to massive ones (like a supercomputer).

4. The Results: What Happened?

The results were a mix of good news and bad news:

The "Formatting" Superpower: The AI models were surprisingly good at following "shape" rules. If you asked them to "write in a JSON format" or "use bullet points," they usually got it right. They are like students who are great at following the layout of a form but struggle with the content.
The "Word" Struggle: The models struggled hard with "word" rules. If you asked them to "include the word 'elephant' exactly three times," they often forgot, used it twice, or used a synonym instead. It's like a student who knows the essay topic but keeps forgetting to write their name on the paper.
The "English Gap": Even the best models performed significantly worse in Indian languages than in English. It's like a musician who can play a concerto perfectly in a major key (English) but fumbles the notes when asked to play in a complex, local scale (Indic languages).
The "Thinking" Boost: They found that when they told the AI to "think step-by-step" before answering (a mode called "Reasoning"), it got much better at following the rules. It's like giving the student a few extra minutes to read the instructions twice before starting the test.
The Winners: The Gemma family of models (from Google) seemed to have the best "multilingual muscle memory," performing closest to English levels. The Proprietary models (the expensive, closed ones) generally did better than the open ones, but the open ones were catching up fast.

5. Why Does This Matter?

This paper is a wake-up call. It shows that while AI is getting smarter, it still has a "cultural blind spot." If we want AI to be truly helpful for billions of people who speak Indian languages, we can't just translate English tests. We need to build tests that understand the rhythm, grammar, and culture of those languages.

In a nutshell: The researchers built a specialized gym for AI to practice following strict rules in Indian languages. They found that while the AI is getting stronger, it still trips over its own feet when the language gets complex or the instructions get specific. But, with the right training (and "thinking" time), it can learn to walk the tightrope.

1. Problem Statement

Instruction-following benchmarks for Large Language Models (LLMs) remain predominantly English-centric. This creates a critical evaluation gap for the hundreds of millions of speakers of Indic languages (a linguistically diverse family characterized by data scarcity, morphological richness, and distinct scripts). While existing multilingual benchmarks exist, they often rely solely on machine translation, which fails to capture cultural nuances, naturalistic constraints, and the specific challenges of low-resource languages. There is a lack of verifiable, rule-based evaluation tools to determine if LLMs can adhere to simple constraints (e.g., word counts, specific keywords, formatting) in non-English Indic contexts.

2. Methodology: Dataset Construction

The authors introduce INDICIFEVAL, a benchmark comprising approximately 800 human-verified examples per language across 14 Indic languages (Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Sanskrit, Tamil, Telugu, and Urdu). The dataset is split into two complementary subsets:

A. INDICIFEVAL-TRANS (Translated & Localized)

Source: Derived from the English IFEval benchmark (Zhou et al., 2023).
Process:
1. Preprocessing: Manual inspection to localize entities (e.g., changing "President of USA" to "Prime Minister of India") and handling constraints requiring English responses (e.g., prepending "Respond in English").
2. Keyword Handling: A specialized pipeline extracts keywords, translates them individually, and re-inserts them into the English prompt before full translation to prevent "translation collapse" (where distinct source words merge into one target word).
3. Full Translation: Prompts are translated using Sarvam-Translate.
4. Verification: Automated regex checks verify constraint descriptions, followed by human verification by native speakers to ensure cultural suitability and semantic validity.
Goal: To enable direct cross-lingual comparison with English baselines.

B. INDICIFEVAL-GROUND (Synthetic & Native)

Motivation: Translated prompts often result in unnatural constraints (e.g., arbitrary keywords or grammatically infeasible sentence structures in Indic languages).
Process:
1. Context Mining: Uses the Sangraha Corpus and TF-IDF to extract high-value, culturally relevant keywords and text segments.
2. Constraint Definition: Focuses on six constraint types identified as problematic in translation: Keyword Inclusion/Prohibition/Frequency, Paragraph Count, Sentence Count, and First Word.
3. Prompt Generation: Uses Gemini-2.5 to generate naturalistic user prompts based on the mined contexts and specific constraints.
4. Verification: Native speakers verify the naturalness and validity of the generated prompts.
Goal: To evaluate instruction following in naturalistic, culturally grounded scenarios that translation alone cannot capture.

3. Evaluation Framework

Models Evaluated: A comprehensive suite of open-weight (Llama 3/4, Gemma 3, Aya, Qwen 3) and proprietary (GPT-5, Gemini 3) models, ranging from 0.5B to 70B+ parameters.
Metrics:
- Prompt-level Accuracy: Percentage of prompts where all constraints are met.
- Instruction-level Accuracy: Percentage of individual instructions met.
- Disparity ( $\Delta$ ): The difference between a model's performance on Indic languages vs. English (lower absolute value indicates better parity).
Verification Pipeline: Modified the original IFEval scripts to integrate the Indic NLP Library for robust sentence segmentation and tokenization specific to Indic scripts.

4. Key Results & Analysis

A. General Performance Gap

English vs. Indic: There is a persistent performance gap. Models consistently perform better on English prompts than on Indic prompts, even for simple rule-based tasks.
High-Resource vs. Low-Resource: Hindi (high-resource) shows the smallest gap, approaching English parity in structural constraints. Sanskrit and Odia (low-resource) show the largest degradation.

B. Model Family Comparisons

Gemma Family: Demonstrates the highest cross-lingual robustness, maintaining the smallest performance gap ( $\Delta < 0.15$ ) across languages, even with smaller parameter counts (27B).
Qwen & Llama: Occupy a competitive middle tier. Qwen often outperforms Llama in specific languages like Bengali and Gujarati.
Aya Family: Shows the highest degradation ( $\Delta > 0.40$ for many languages), struggling to transfer instruction-following precision despite its multilingual focus.
Proprietary vs. Open-Weight: Proprietary models (GPT-5, Gemini 3) maintain a significant absolute accuracy advantage (10+ points) over open-weight models, though open-weight models (specifically Gemma-27B) narrow the gap in high-resource scenarios like Hindi.

C. Constraint-Specific Insights

Formatting: Models excel at structural constraints (JSON, bullet points, capitalization).
Lexical Tasks: Models struggle significantly with keyword inclusion (positive constraints) compared to keyword exclusion (negative constraints). They can suppress concepts better than they can generate precise lexical forms.
Cross-Lingual Tasks: When prompted in an Indic language but asked to respond in English, performance drops significantly, especially for Llama and Aya families.

D. Impact of Reasoning ("Thinking")

Using Qwen3 models, the study found that enabling "Thinking" modes consistently reduces the performance gap between English and Indic languages across all parameter sizes.
This suggests that additional reasoning steps help models better bridge the cross-lingual gap, though sub-billion models lack the capacity to leverage this effectively.

5. Significance & Contributions

First Comprehensive Indic Benchmark: Provides the first large-scale, verifiable instruction-following benchmark for 14 Indic languages, addressing a major gap in multilingual LLM evaluation.
Dual-Subset Approach: By combining TRANS (parallel, comparable to English) and GROUND (native, naturalistic), the benchmark offers a holistic view of model capabilities, distinguishing between translation artifacts and genuine language understanding.
Methodological Framework: Introduces a robust pipeline for creating verifiable benchmarks in low-resource languages, including strategies to handle translation collapse and cultural localization.
Empirical Insights: Reveals that while formatting is mastered, lexical precision and cross-lingual grounding remain major bottlenecks. It highlights that model size alone does not solve the gap; specific training objectives (like Gemma's) and reasoning mechanisms are crucial.
Open Release: The dataset and evaluation scripts are released to accelerate research in multilingual constrained generation.

Conclusion

The paper concludes that while LLMs are making progress, instruction-following in the broader Indic family still lags significantly behind English. High-resource languages like Hindi are nearing parity, but low-resource languages and complex lexical constraints remain challenging. The study emphasizes that future improvements require not just larger models, but better localization strategies and reasoning capabilities tailored to the morphological and syntactic complexities of Indic languages.