💬 NLP

IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text

IndiaFinBench is the first publicly available benchmark designed to evaluate large language models on Indian financial regulatory text, featuring 406 expert-annotated question-answer pairs from SEBI and RBI documents that reveal significant performance gaps among models, particularly in numerical reasoning, while substantially outperforming non-specialist human baselines.

Original authors: Rajveer Singh Pall

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rajveer Singh Pall

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot librarian who has read almost every book in the world. You ask it a question about a library in New York, and it answers perfectly. But then you ask it a question about a very specific, old, and complicated library in India, full of unique rules and local slang. Suddenly, the robot stumbles. It doesn't know the local customs, and it gets confused by the specific way Indian laws are written.

This paper introduces IndiaFinBench, a new "test" designed to see how well these AI robots can handle the specific, tricky world of Indian financial regulations.

Here is the story of the paper, broken down into simple parts:

1. The Problem: The "Western Bias"

Until now, all the tests for AI in finance were like driving tests done only on American highways. They used US laws, US bank reports, and US news.

The Reality: India has its own financial "highways" run by two main traffic cops: SEBI (for the stock market) and RBI (for the central bank).
The Issue: Indian financial rules are messy. They are full of numbers hidden in paragraphs, rules that change over time (like a recipe that gets updated every year), and special words that only Indian experts know. Existing AI tests didn't check if the robots could handle this specific mess.

2. The Solution: The "Indian Driving Test"

The authors built IndiaFinBench, a brand-new exam with 406 questions.

Where did the questions come from? They dug through 192 official documents from SEBI and RBI, ranging from 1992 to 2026.
What kind of questions?
- The "What does this mean?" test: Can the AI read a rule and tell you exactly what a bank is allowed to do?
- The "Math" test: Can the AI do the math hidden in the text? (e.g., "If a bank has this much profit, how much dividend can they pay?")
- The "Lie Detector" test: Can the AI spot if two different rules contradict each other?
- The "Time Travel" test: Can the AI figure out which rule was active in 2015 versus 2023, given that rules often get updated?

3. The Exam Takers: 12 Different Robots

The researchers put 12 different AI models (the "robots") through this test. Some were huge giants (like Gemini 2.5 Flash and LLaMA-3.3-70B), and some were smaller, lighter models (like Gemma 4).

The Rules: The robots had to answer using only the text provided in the question. They couldn't use their "memory" of the internet. This ensures they are actually reading and understanding, not just guessing.

4. The Results: Who Passed?

The results were surprising and revealed three distinct groups:

The Top Tier (The A-Students):
- Gemini 2.5 Flash came out on top with about 90% accuracy. It was the best at reading the rules and doing the math.
- Qwen3-32B and LLaMA-3.3-70B were right behind it, almost tied.
- The Surprise: Llama 4 Scout 17B (a much smaller robot) performed just as well as the massive 70B robot. It's like a compact car getting the same gas mileage as a massive truck. This suggests that how the robot is trained matters more than just how big it is.
The Middle Tier (The C-Students):
- Models like GPT-OSS and Mistral hovered around 75-79%. They were okay, but they made mistakes on the harder math and time-travel questions.
The Bottom Tier (The F-Student):
- Gemma 4 E4B (a very small model) scored around 70%. It struggled the most, especially with math and understanding the timeline of rules.

The Human Baseline:
Even a human who isn't a finance expert only scored 60%. This proves the test is actually hard! The best AI (Gemini) beat the average human by a huge margin, but the worst AI was barely better than a non-expert human.

5. Where Did the Robots Fail? (The Error Analysis)

Even the smartest robots had trouble spots:

The "Time Travel" Problem: The hardest part for almost everyone was Temporal Reasoning. Indian rules often say, "This rule from 2010 is now replaced by the 2022 rule, unless you fall under the 2015 exception." The robots got confused about which rule applied when.
The "Math" Problem: Numerical Reasoning was the biggest differentiator. The gap between the best and worst robot on math was huge (over 35%).
The "Knowledge" Problem: Smaller robots often failed because they simply didn't know what words like "AIF" or "FEMA" meant. They were guessing.

6. Why This Matters

This paper is a wake-up call. It shows that:

One size does not fit all: An AI trained on US laws isn't automatically good at Indian laws.
Bigger isn't always better: A smaller, well-trained model can beat a giant one on specific local tasks.
We need local tests: If we want AI to help Indian banks, lawyers, and regulators, we need to test them on Indian data, not just American data.

The Bottom Line:
The authors have released this test, the questions, and the answers to the public. They want everyone to keep testing new robots on Indian rules so we can build AI that truly understands the Indian financial world, not just the Western one. It's like giving the robots a local driver's license instead of just a US one.

1. Problem Statement

While Large Language Models (LLMs) have shown proficiency in general reasoning and Western financial contexts (e.g., SEC filings, US earnings reports), there is a significant gap in evaluating their performance on non-Western regulatory frameworks, specifically India's.

Unique Challenges: Indian financial regulation (governed by SEBI and RBI) involves dense numerical thresholds, complex chains of superseding circulars requiring temporal reasoning, and jurisdiction-specific terminology (e.g., LODR, PMLA, SFB) that Western-trained models often fail to interpret correctly.
Lack of Benchmarks: Existing financial NLP benchmarks (FinQA, FinanceBench, etc.) rely exclusively on US/European data, leaving the Indian regulatory domain uncharacterized and hindering the deployment of reliable AI in Indian compliance and legal-tech sectors.

2. Methodology: IndiaFinBench Construction

The authors constructed IndiaFinBench, the first publicly available benchmark for this domain, through the following steps:

Data Source: 192 primary regulatory documents (circulars, master directions, policy statements) from the Securities and Exchange Board of India (SEBI) and the Reserve Bank of India (RBI), spanning 1992–2026.
Dataset Composition: 406 expert-annotated question-answer (QA) pairs categorized into four distinct reasoning tasks:
1. Regulatory Interpretation (REG, 174 items): Identifying rules, thresholds, and scope of applicability.
2. Numerical Reasoning (NUM, 92 items): Performing arithmetic on embedded figures (e.g., calculating dividend limits based on capital ratios).
3. Contradiction Detection (CON, 62 items): Determining if two regulatory passages contradict each other (Yes/No).
4. Temporal Reasoning (TMP, 78 items): Establishing chronological order of events or identifying which rule version was active at a specific time.
Annotation & Validation:
- Primary Annotation: Conducted by a domain expert.
- Model-Based Validation: A secondary pass using LLaMA-3.3-70B to verify unambiguity (90.7% agreement).
- Human Inter-Annotator Agreement (IAA): A second human annotator validated 60 random items, achieving substantial agreement ( $\kappa = 0.611$ for contradiction detection) and 76.7% overall agreement.
Difficulty Levels: Items were stratified into Easy (39.4%), Medium (44.8%), and Hard (15.8%) based on reasoning complexity.

3. Experimental Setup

Models Evaluated: 12 contemporary LLMs ranging from 4B to 1T parameters (including Gemini 2.5 Flash/Pro, Qwen3-32B, LLaMA-3.3-70B, Llama 4 Scout, GPT-OSS, DeepSeek R1, and Mistral).
Conditions: Strict zero-shot evaluation with context-only constraints (no external knowledge allowed) to test reasoning rather than memorization.
Scoring: A multi-stage matching pipeline including exact match, fuzzy token matching (RapidFuzz), and numerical extraction validation.
Baseline: A non-specialist human baseline (30 items, 60.0% accuracy) was established for comparison.

4. Key Results

Overall Performance: Models significantly outperformed the non-specialist human baseline (60.0%).
- Top Performer: Gemini 2.5 Flash achieved the highest overall accuracy (89.7%).
- Lowest Performer: Gemma 4 E4B scored 70.4%.
- Human Baseline: 60.0% (95% CI: [42.3%, 75.4%]).
Performance Tiers: Bootstrap significance testing (10,000 resamples) revealed three distinct tiers:
1. Tier 1 (83–90%): Gemini 2.5 Flash, Qwen3-32B, LLaMA-3.3-70B, Llama 4 Scout 17B, Kimi K2.
2. Tier 2 (75–79%): LLaMA-3-8B, GPT-OSS models, Gemini 2.5 Pro, Mistral-7B, DeepSeek R1 70B.
3. Tier 3 (~70%): Gemma 4 E4B.
Task-Specific Insights:
- Numerical Reasoning (NUM): The most discriminative task (35.9% spread between best and worst). Gemini 2.5 Flash (84.8%) vastly outperformed Gemini 2.5 Pro (48.9%), the latter penalized for verbose outputs.
- Contradiction Detection (CON): The most uniform task (91.0% average), with Llama 4 Scout 17B reaching 98.4%.
- Temporal Reasoning (TMP): A major failure point for top models; Gemini 2.5 Flash (88.5%) and Qwen3-32B (92.3%) led, while DeepSeek R1 70B struggled (70.5%).
Scaling Anomalies:
- Parameter Efficiency: Llama 4 Scout 17B performed statistically indistinguishably from LLaMA-3.3-70B (83.3% vs 83.7%), suggesting training quality outweighs raw parameter count in this domain.
- Reasoning Architecture Failure: DeepSeek R1 70B, despite its chain-of-thought design, ranked 11th (75.1%), indicating that explicit reasoning chains do not automatically translate to success in tracking complex regulatory amendment timelines.
- Scaling Plateau: GPT-OSS 120B (77.1%) and GPT-OSS 20B (76.8%) showed no statistically significant difference.

5. Error Analysis

The authors classified errors into four categories:

Domain Knowledge Failure (DKF): Misunderstanding Indian terminology (dominant in smaller models like Gemma 4 E4B at 43%).
Numerical Reasoning Failure (NRF): Arithmetic or unit conversion errors.
Temporal Reasoning Failure (TRF): Incorrect ordering of events or rule versions (dominant in top models, e.g., 40–49% of errors for Gemini and DeepSeek).
Context Grounding Failure (CGF): Hallucinating external knowledge (rare, <3% across all models).

Key Finding: Top-tier models possess adequate domain knowledge but fail at complex temporal reasoning chains. Smaller models fail at both knowledge and reasoning.

6. Significance and Contributions

First Public Benchmark: Establishes the first open resource for evaluating LLMs on Indian financial regulation, addressing a critical gap in global NLP evaluation.
Methodological Rigor: Introduces a dual-layer validation (model-based + human IAA) and robust statistical significance testing (paired bootstrap) to define performance tiers.
Practical Implications:
- Highlights that model size is not the sole predictor of performance in specialized regulatory domains; efficient architectures (Llama 4 Scout) can outperform larger predecessors.
- Identifies Temporal Reasoning and Numerical Reasoning as the primary bottlenecks for current LLMs in this domain.
- Provides actionable insights for developers: Top models struggle with tracking amendment chains, while smaller models lack necessary domain vocabulary.
Open Science: The dataset, evaluation code, and all model outputs are publicly released to foster research in multilingual and jurisdiction-specific financial NLP.

Conclusion

IndiaFinBench demonstrates that while current frontier LLMs can outperform non-expert humans on Indian regulatory tasks, they still face significant challenges in temporal reasoning and complex numerical inference. The benchmark reveals that the "best" model for this domain is not necessarily the largest, but rather one with specific architectural strengths or high-quality instruction tuning, and that current reasoning-specialized models (like DeepSeek R1) may not yet solve the unique structural problems of Indian regulatory text.