💻 computer science

How Reliable Are Bengali LLM Benchmarks? A Cross-Lingual Contamination and Robustness Audit of Open-Weight Language Models

This paper introduces and validates a systematic framework for auditing the reliability of Bengali LLM benchmarks by detecting cross-lingual data contamination and measuring robustness against perturbations, revealing that initial pilot results show performance barely above chance levels.

Original authors: Md Fahim Faisal Tanha

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Md Fahim Faisal Tanha

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to judge how good a new student is at solving math problems. You give them a test, they get a high score, and you declare them a genius. But what if, before the test, they had secretly memorized the answer key? Or what if they had seen the same problems in a different language and just guessed the answers based on that? This is the tricky world of Artificial Intelligence (AI) testing. Scientists build "Large Language Models" (LLMs)—super-smart computer programs that can read, write, and chat in many languages. To see if these programs are actually smart or just good at memorizing, researchers give them standardized tests called "benchmarks."

However, there is a sneaky problem called "contamination." It's like if a teacher accidentally left the answer key on the teacher's desk, and the student found it before the exam. The student gets a perfect score, but they didn't actually learn the material; they just memorized the answers. This is a huge deal because if the tests are "contaminated," we can't trust the scores. We might think an AI is great at a language like Bengali (spoken by hundreds of millions of people) when it's actually just having seen the English version of the test before.

Now, let's look at a new study that acts like a detective, investigating whether these AI tests for the Bengali language are fair or if they are full of hidden leaks.

The Detective Work: Auditing Bengali AI Tests

The researchers in this paper are asking a very important question: Are the scores we see for AI models on Bengali tests real, or are they inflated because the models had prior exposure?

They focused on Bengali because it's a massive language that has only recently started getting its own AI tests. Many of these tests are just direct translations of famous English tests. The team worried that if an AI model had seen the English version of a question during its training, it might be able to guess the Bengali answer correctly without actually understanding Bengali. This is called "cross-lingual contamination."

To solve this mystery, the researchers built a special "audit framework"—a set of tools to check for prior exposure. They used three main methods:

The "Likelihood" Check: They looked at how surprised the AI was by the test questions. If the AI had seen the question before, it wouldn't be surprised; it would be very confident. If it was seeing it for the first time, it would be more hesitant.
The "Fill-in-the-Blank" Test: They tried to trick the AI by hiding one of the wrong answers in a multiple-choice question and asking it to guess what the missing option was. If the AI could guess the exact missing word, it likely had the whole test memorized.
The "Ground Truth" Search: For some models that are completely open (where we can see exactly what data they were trained on), they literally searched the training data to see if the test questions were there.

They also tested robustness. This is like changing the wording of a question slightly—swapping a number or adding a silly, irrelevant sentence—to see if the AI still gets it right. If the AI's score crashes when you change the question slightly, it suggests the AI wasn't really "thinking"; it was just memorizing the exact pattern.

What They Found (So Far)

The researchers didn't just talk about the theory; they actually ran a small "pilot" test to see if their detective tools worked. They used a smaller AI model (called Qwen2.5-1.5B) and tested it on 500 questions from a Bengali reading comprehension test called BoolQ-bn.

Here is what the pilot revealed:

The Score Was Barely Better Than a Coin Flip: The model got an accuracy of 0.568. Since this is a yes/no question test, a random guess would get you 0.50. The model only did about 7 points better than pure luck. This suggests that at this size, the model doesn't have a strong grasp of Bengali reading comprehension yet.
The "Prior Exposure" Test Was Inconclusive: They tried to see if the model had memorized the answers using their likelihood tools, but they needed a special "control" set of questions (questions the model definitely hadn't seen) to compare against. That part is still being finished.
The "Distraction" Test: They added a distracting sentence to the questions to see if the model would get confused. The model's score dropped slightly to 0.558, but the difference wasn't statistically significant (the math says it could just be random noise). This means the pilot was too small to say for sure if the model was fragile or not.

The Big Picture

The main takeaway from this paper isn't a final verdict on whether Bengali AI is exhibiting prior exposure or not. Instead, the paper is a blueprint for how to find out.

The researchers have built a complete, open-source toolkit that anyone can use to audit these tests. They have listed all the available Bengali benchmarks, created a protocol to check for contamination, and shown that it is possible to run these expensive tests on free computer resources (like Google Colab).

They are currently running the full audit on many more models and benchmarks. The pilot proved the system works, but the full results are still being calculated. The paper argues that until we run these audits, we can't be sure if the high scores we see for AI in Bengali are real intelligence or just the result of the models having seen the English answers before.

In short, this paper is a call to action: "Let's stop guessing if AI is smart in Bengali and start using these detective tools to prove it." They are making sure that when we say an AI is good at a language, it's because it actually understands the language, not because it memorized the test.

Technical Summary: Reliability of Bengali LLM Benchmarks

1. Problem Statement

Bengali, spoken by approximately 250 million people, has recently seen a proliferation of Large Language Model (LLM) evaluation benchmarks. However, the reliability of these benchmarks is compromised by two primary threats:

Benchmark Data Contamination (BDC): Evaluation items frequently leak into model training corpora. Crucially, this contamination is cross-lingual; models trained on English source texts can achieve inflated scores on their Bengali translations without ever seeing the translated text. Since most Bengali benchmarks are direct translations of English test sets, they are structurally vulnerable to this risk.
Lack of Robustness: Performance on low-resource language benchmarks may reflect surface-level memorization rather than genuine capability. Recent evidence suggests that scores can collapse under semantically equivalent perturbations, yet this fragility has not been systematically audited for Bengali.

Despite the rapid adoption of these benchmarks for deployment decisions in education and public services, no prior study has audited Bengali benchmarks for contamination or robustness. This leaves the validity of reported performance scores in question.

2. Methodology

The author presents a systematic audit framework designed to evaluate the reliability of Bengali LLM benchmarks. The framework is applied to open-weight models (1B–9B parameters) to leverage access to token-level likelihoods and, in some cases, full pre-training corpora.

2.1 Audit Framework Components

The framework employs a triangulation of three detection families:

Likelihood-based Signals (Min-K% Prob): Measures the average log-probability of the $k\%$ least likely tokens for benchmark items. Items present in training data typically exhibit higher scores. This is compared against a private, never-published probe set and a temporal control (Pythia models trained before the benchmarks existed).
Generation-based Tests (TS-Guessing): For multiple-choice questions, the model is prompted to reconstruct a masked incorrect option. Accuracy significantly above a chance baseline indicates memorization of the test file rather than task understanding.
Ground-Truth Corpus Search: For models with fully public pre-training corpora (Tier B: OLMo, Pythia, BLOOM), the framework performs exact 13-gram overlap searches against the training data (Dolma, The Pile, ROOTS) in both Bengali and English forms.

2.2 Cross-Lingual and Robustness Analysis

Cross-Lingual Contamination (RQ2): Utilizing the parallel Bengali-English design of BEnQA, the study compares memorization signals and accuracy between English source items and their Bengali counterparts. It tests whether stronger memorization on the English source predicts correct answers on the Bengali translation.
Robustness Audit (RQ3): The framework applies meaning-preserving perturbations to benchmark items, including:
- Option reordering (for MCQs).
- Distractor sentence insertion.
- Numeric substitution (for math problems).
- Round-trip machine translation paraphrasing.
- The study measures the delta in accuracy and the stability of model rankings under these perturbations.

2.3 Experimental Setup

Models: Audited models include Llama-3.2, Qwen2.5, Gemma-2, Mistral, TituLLM, TigerLLM, OLMo, Pythia, and BLOOM.
Benchmarks: A curated inventory of 8 public Bengali benchmarks (e.g., BEnQA, Bangla MMLU, BanglaMATH, BoolQ-bn), distinguishing between native (authored in Bengali) and translated (derived from English) sources.
Compute: All experiments were conducted on free-tier GPU compute (Kaggle/Google Colab) using 4-bit quantization for larger models.
Pilot: A completed pilot evaluated Qwen2.5-1.5B-Instruct on a 500-item subsample of BoolQ-bn.

3. Key Contributions

Curated Inventory: A versioned, documented inventory of public Bengali LLM benchmarks, detailing provenance, size, licensing, and construction methods.
Contamination-Audit Protocol: The first protocol specifically for Bengali benchmarks, combining likelihood, generation, and corpus-search methods.
Cross-Lingual Test: A novel experimental design using parallel data (BEnQA) to isolate and measure cross-lingual contamination.
Robustness Instrument: A perturbation suite to measure score stability and ranking shifts, linking fragility to potential contamination.
Public Release: All code, data manifests, and per-item audit outputs are publicly released to enable community re-auditing.

4. Pilot Results

The pilot study on Qwen2.5-1.5B-Instruct (500 items from BoolQ-bn) yielded the following:

Accuracy: 0.568 (95% CI [0.524, 0.610]), which is barely above the 0.50 chance level for a binary task. The model outperformed the majority-class baseline by only 3.4 percentage points.
Robustness: A distractor perturbation shifted accuracy by -1.0 percentage point (0.568 to 0.558). The McNemar test on discordant items was not statistically significant ( $p=0.302$ ), though the author notes the pilot sample size was underpowered for detecting small deltas.
Contamination Signals: The mean Min-K% Prob was -3.767. Full AUROC scores for contamination detection are pending the comparison with the private probe set.
Feasibility: The pilot confirmed that a full model $\times$ benchmark audit cell (accuracy + Min-K% + perturbation) can be executed in approximately 0.56 GPU-hours on a single T4 GPU.

5. Significance and Claims

The paper positions itself as the first systematic audit of Bengali LLM benchmarks. Its significance lies in:

Foundational Question: It challenges the interpretation of high scores on Bengali benchmarks, asking whether they reflect genuine capability or memorization of leaked data.
Methodological Rigor: By combining three distinct detection families and using a temporal control (Pythia), the study aims to provide stronger evidence than single-method approaches.
Scalability: The framework is designed to be language-agnostic and computationally cheap enough (free-tier compatible) to be applied to other mid-sized, translation-dependent languages.
Community Utility: The release of the protocol and inventory allows the community to continuously monitor the reliability of Bengali benchmarks as new models and datasets emerge.

The author explicitly states that the full audit grid (covering all models and benchmarks) is currently in progress, and this manuscript reports the framework, inventory, protocol, and the completed pilot. They maintain a modest stance, noting that likelihood-based detectors provide evidence rather than definitive proof for open-weight models and that the absence of a signal does not guarantee a benchmark is clean.