Memory or Generalization? Temporal Probing of Data Contamination in Bengali LLM Benchmarks
This paper introduces "temporal probing," a direct validity test using difficulty-matched, post-cutoff items to measure data contamination in Bengali LLM benchmarks, finding that current models on the BoolQ-bn dataset show no significant inflation from memorization despite the prevalence of static, translated test sets.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of artificial intelligence, researchers rely on tests to measure how smart a computer program is. These tests, called benchmarks, are like standardized exams for machines. For a long time, these exams were written in English. As the technology has grown, developers have translated these English tests into other languages to see how well their programs understand the world's many tongues. One of the most important languages to test is Bengali, spoken by roughly a quarter of a billion people. However, a shadow hangs over these translated tests. Because the internet is so vast, the text of these exams often ends up inside the massive libraries of data that teach these programs how to speak. If a program has already read the exam questions during its training, it isn't truly showing off its intelligence when it answers them correctly; it is simply remembering the answers it saw before. This problem, known as data contamination, makes it difficult to know if a high score means the program is actually capable or just has a good memory.
A researcher named Md Fahim Faisal Tanha set out to solve this mystery for Bengali language models. Instead of trying to guess whether a model had seen a question, he used time as a tool to find the truth. He realized that if a model was trained on data up to a certain date, it could not possibly know about anything published after that date. To test this, he took the old, existing Bengali exam questions and created brand-new, difficult-to-match questions using news articles and test papers published in 2025 and 2026. These new questions were carefully crafted to be just as hard as the old ones, but they were guaranteed to be unseen by the models because they were written after the models stopped learning. By comparing how well the models did on the old questions versus the new ones, he could see if the old scores were inflated by memory.
The study focused on seven different open-source language models, ranging from small to medium sizes, and tested them on eighty-four pairs of questions. The results were surprisingly reassuring for the community. For most of the models, the scores on the old questions were almost exactly the same as the scores on the new, unseen questions. This suggests that the models were not memorizing the old exams; they were actually using their understanding of the language to solve the problems. The largest difference found was a tiny gap of about eight percentage points, which statistical analysis showed was likely just random noise rather than a sign of data contamination. In other words, the current scores for these Bengali models appear to be trustworthy.
There was one interesting exception that taught the researchers a valuable lesson about how to design these tests. One model, a large version of the Qwen family, actually performed significantly better on the brand-new questions than on the old ones. At first, this seemed strange, but the researchers realized it wasn't a sign of contamination. Instead, it meant the new questions were simply easier for that specific model to answer than the old, translated questions. The old questions required complex reasoning that the model struggled with, while the new questions were more straightforward facts. This gap shrunk as the researchers refined their matching process, proving that the initial difference was a flaw in the test design, not a flaw in the model's performance. This finding highlights that the new method of using time to check for data contamination is powerful enough to spot not just memory, but also subtle mismatches in difficulty.
Ultimately, this work provides a practical way for the Bengali artificial intelligence community to verify their results without needing expensive tools or complex guessing games. By using a simple, free method that compares old questions with new ones, researchers can now be more confident that their high scores reflect genuine intelligence. The study concludes that for the models tested, the benchmarks are valid and the scores are real. As newer and more powerful models arrive in the future, this same approach can be used to ensure that the progress being reported is based on true capability, keeping the field honest and moving forward on solid ground.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.