CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

This paper introduces CricBench, the first multilingual Text-to-SQL benchmark for cricket analytics, which evaluates seven leading LLMs across four cricket formats and four languages, revealing that while models achieve high syntactic accuracy, they struggle significantly with semantic correctness and domain-specific nuances.

The Gist

Imagine you have a super-smart robot librarian who has read every book in the world. You ask it, "Who won the last game?" and it answers perfectly. You ask, "What was the weather like?" and it's spot on.

But then, you ask it a very specific question about a cricket match: "Show me the average speed of the bowler who played for India in the 2011 World Cup, but only for the overs where it rained."

Suddenly, the robot librarian starts stuttering. It gives you a perfectly formatted answer, but the numbers are wrong. It's like a chef who can perfectly chop vegetables and plate a dish (the syntax), but forgets to actually cook the meat (the logic), serving you a raw, inedible steak that looks beautiful.

This is exactly what the paper CricBench is about.

The Big Idea: The "Cricket Brain" Test

The authors created a special test called CricBench. Think of it as a "driver's license exam" for Artificial Intelligence, but instead of driving a car, the AI has to drive a cricket database.

Cricket is a sport with billions of fans, but it's also incredibly complex. It has different formats (Test matches that last 5 days, T20s that last 3 hours, and the IPL, which is like a high-stakes franchise league). The data is messy, the rules are specific, and fans often ask questions in different languages (English, Hindi, Punjabi, Telugu) or mix them up (like saying, "What is the Strike Rate of Virat?").

The researchers wanted to see: Can our best AI models actually understand cricket, or are they just guessing?

How They Tested It

They didn't give the AI a cheat sheet. They didn't say, "Here is a formula for calculating runs." They just gave the AI the blueprint of the database (the list of tables and columns) and a simple question in a human language.

It's like handing a mechanic a car engine diagram and saying, "Fix the noise," without telling them which part is broken.

The Shocking Results

The results were a bit of a reality check for the AI world:

1. The "Perfectly Wrong" Problem: The AI models were great at writing code that looked right. They could write a SQL query (the language databases speak) that ran without crashing 99% of the time. But when they ran that code, the answer was usually wrong.

   • Analogy: It's like a student who writes a math equation perfectly on the blackboard but gets the final number wrong because they forgot to multiply by 2. The teacher says, "Great handwriting, but wrong answer."

2. No Single Champion: Just like in cricket, no single AI model was the best at everything.

   • One model was great at "Test" matches (the long, slow games).
   • Another model was better at "IPL" (the fast, flashy franchise games).
   • Some models completely failed at "ODI" (the one-day games), getting 0% of the hard questions right.

3. The Language Barrier: The researchers tested the AI in English, Hindi, Punjabi, and Telugu. Surprisingly, the AI didn't get confused by the languages. If it was bad at cricket in English, it was equally bad in Hindi. The problem wasn't the language; it was the logic.

4. The "Generalist" Trap: The paper compared these AI models to how they perform on general business questions (like "How many sales did we make last month?").

   • The Gap: On general questions, the AI was about 60% accurate. On cricket questions, that accuracy plummeted to under 15%.
   • Analogy: Imagine a world-class chess player who can beat anyone at chess. But if you ask them to play a game of Go (a different board game), they might lose to a beginner. Being smart at "general stuff" doesn't mean you are smart at "specialized stuff."

Why Does This Matter?

The paper concludes that current AI is like a tourist with a phrasebook. It can point to a picture of a "bowler" and say the word, but it doesn't understand the rules of the game.

To make AI truly useful for sports analysts, doctors, or financial experts, we can't just make the AI "bigger" or "smarter" in a general sense. We need to teach it the specific rules and logic of those fields.

The Takeaway

CricBench is a wake-up call. It shows that while AI is amazing at writing code and chatting, it still struggles to be a true "expert" in complex, real-world domains like cricket. Until we fix this "logic gap," AI will remain a helpful assistant that needs a human expert to double-check its homework.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have achieved significant success in general Text-to-SQL tasks (e.g., WikiSQL, Spider, BIRD), their ability to handle highly specialized domains remains under-explored. Cricket analytics presents a unique challenge due to:

• Domain Complexity: It requires granular reasoning involving precise mathematical operations (e.g., calculating Economy Rate), temporal awareness, compound derived metrics, and phase-specific filters.
• Entity Resolution: Dynamic player affiliations across eras and teams (national vs. franchise) create complex mapping challenges.
• Multilingual Needs: The cricket fanbase, particularly in India, is linguistically diverse, often using code-mixed dialects (e.g., English technical terms within Hindi/Punjabi/Telugu sentences).
• The Gap: Current benchmarks are predominantly English-centric and lack the depth required for specialized sports analytics. Existing models often generate syntactically valid SQL that fails semantically when applied to domain-specific logic.

2. Methodology

Dataset Construction (CricBench)

• Data Source: Comprehensive ball-by-ball records from Cricsheet, transformed into strict SQLite schemas.
• Scope: Covers four cricket formats:
  • International: Test, ODI, T20I (sharing a common schema of 5 core tables: Matches, Deliveries, Players, PlayerInMatch, FielderDismissals).
  • Domestic: Indian Premier League (IPL) (distinct schema tailored to franchise-based structures).
• Scale:
  • 633 unique base queries manually authored by domain experts.
  • 2,654 total evaluation instances after translation into four languages: English, Hindi, Punjabi, and Telugu.
  • Difficulty Levels: Categorized as Easy, Medium, and Hard based on SQL complexity (joins, aggregations, temporal filtering, derived metrics).
• Gold Standard: Queries and SQL answers were manually verified against third-party records (ESPNcricinfo, Cricbuzz) to ensure factual correctness.

Evaluation Protocol

• Prompting Strategy: Schema-only prompting. Models receive only the database schema and minimal instructions (e.g., definition of a "legal delivery"). No few-shot examples, retrieval augmentation, or domain-specific context injection is provided. This isolates the model's intrinsic reasoning capabilities.
• Models Evaluated: Seven state-of-the-art models, including:
  • Proprietary: GPT-5 Mini, Claude Sonnet 4.
  • Open-Weight: DeepSeek R1 & V3, Qwen 235B, Llama 3.1 (8B), Gemma 2 (9B).
• Metrics:
  • Execution Accuracy (EX): Percentage of queries that run without syntax/runtime errors.
  • Data Match Accuracy (DMA): Percentage of queries returning results that exactly match the Gold Standard (the primary metric for semantic correctness).

3. Key Contributions

1. First Cricket-Specific Benchmark: CricBench is the first expert-curated, multilingual Text-to-SQL benchmark specifically for cricket analytics.
2. Multilingual & Code-Mixed Support: Introduces verified test sets in Hindi, Punjabi, and Telugu, addressing the lack of Indic language resources in specialized SQL benchmarks and simulating real-world code-mixed user behavior.
3. Intrinsic Capability Assessment: By avoiding domain-specific context injection, the benchmark reveals the raw reasoning limits of LLMs, showing that even frontier models struggle with specialized queries.
4. Domain Gap Quantification: Provides empirical evidence of the performance drop when moving from general-purpose benchmarks (BIRD) to specialized domains.

4. Key Results

Performance Overview

• No Single Dominator: Model rankings shift drastically across formats.
  • GPT-5 Mini leads on Test (12.4% DMA) and ODI (10.9%).
  • Qwen 235B leads on T20I (17.5%) and IPL (28.7%).
  • DeepSeek R1 achieves a high 33.0% on Test but drops to 4.5% on T20I.
• The "Execution-Accuracy Disconnect": A critical finding is the gap between syntactic validity and semantic correctness.
  • Models achieve >98% Execution Accuracy (queries run successfully).
  • However, Data Match Accuracy (DMA) is extremely low, with the best models achieving <29% DMA.
  • Example: GPT-5 Mini has 99.7% EX on Test but only 12.4% DMA.
• Hard Queries: All models scored 0% DMA on hard ODI queries, indicating a total failure in complex domain reasoning for this specific format.
• Small Models: Smaller models (Llama 3.1 8B, Gemma 2 9B) degrade severely on complex formats (T20I/IPL), often achieving near-zero DMA.

Multilingual Analysis

• Performance is uniformly low across all four languages (DMA < 15% for most models).
• Code-mixing (e.g., using English terms like "Strike Rate" in Hindi sentences) did not significantly degrade performance compared to pure English, suggesting the bottleneck is domain reasoning, not language understanding.

The Domain Gap (CricBench vs. BIRD)

• Comparing CricBench results against the general-purpose BIRD benchmark reveals a massive domain gap.
• Models lose 37–55 percentage points in accuracy when moving from general business queries to cricket analytics.
• Counterintuitive Finding: The models with the highest general performance (e.g., GPT-5 Mini) suffered the largest absolute drops (54.9 points), suggesting that general Text-to-SQL proficiency does not transfer proportionally to specialized domains.

5. Error Analysis

Qualitative analysis of failures revealed four main categories:

1. Schema Hallucination: Models frequently referenced non-existent columns (e.g., total_runs) despite being provided the correct schema, relying on parametric knowledge of "typical" databases.
2. Derived Metric Errors: Incorrect computation of domain-specific formulas (e.g., failing to filter for legal deliveries when calculating Economy Rate).
3. Entity Resolution Failures: Inability to correctly map players to teams across different eras or franchises.
4. Temporal Logic Errors: Misinterpreting time-based constraints (e.g., "last 5 matches") by using incorrect ordering or limiting logic.

6. Significance and Conclusion

• Current State of AI: The study demonstrates that current LLMs, even frontier models, fundamentally lack the domain reasoning required for specialized analytics. They can generate syntactically correct SQL but fail to capture the semantic logic of the domain.
• Implications: General-purpose scaling and instruction tuning are insufficient for specialized domains. Future work must focus on domain-specific fine-tuning, structured Retrieval-Augmented Generation (RAG) with rule injection, and reasoning-enhanced architectures.
• Resource: CricBench establishes a challenging baseline and provides a publicly available dataset and evaluation harness to drive research in specialized Text-to-SQL generation.