💬 NLP

BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks

The paper introduces BenchMarker, an education-inspired toolkit that uses LLM judges to detect contamination, shortcuts, and writing errors in multiple-choice benchmarks, revealing that such flaws persist across 12 major datasets, significantly distort evaluation metrics, and often emerge even in previously repaired benchmarks.

Original authors: Nishant Balepur, Bhavya Rajasekaran, Jane Oh, Michael Xie, Atrey Desai, Vipul Gupta, Steven James Moore, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nishant Balepur, Bhavya Rajasekaran, Jane Oh, Michael Xie, Atrey Desai, Vipul Gupta, Steven James Moore, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher preparing a final exam for your students. You want the test to be fair, clear, and actually measure what the students have learned, not just how well they can cheat or guess.

Now, imagine that in the world of Artificial Intelligence (AI), researchers are doing the exact same thing. They create "exams" (called benchmarks) to test how smart their AI models are. But, according to this paper, many of these AI exams are broken. They are full of typos, confusing questions, and some questions are so easy to find online that the AI is just "cheating" by memorizing the answers instead of thinking.

The authors of this paper, a team of researchers from universities and companies like Scale AI, built a new tool called BenchMarker. Think of BenchMarker as a super-strict, education-savvy teaching assistant whose only job is to grade the exams before the AI takes them.

Here is how BenchMarker works, using three simple ways to spot a bad exam question:

1. The "Cheating" Check (Contamination)

The Problem: Imagine you give a student a math problem, but that exact problem was posted on a popular homework-help website last week. If the student Googles it, they get the answer instantly. They didn't learn math; they just found the answer key.
The BenchMarker Fix: BenchMarker acts like a detective. It takes every question on the AI exam and searches the entire internet. If it finds the question (or the answer) sitting on a website, it flags it as "Contaminated."

The Result: They found that nearly half of the questions in one famous test (TruthfulQA) were already online. The AI wasn't being smart; it was just remembering what it saw on the internet.

2. The "Trick Question" Check (Shortcuts)

The Problem: Imagine a multiple-choice question where the question asks, "Which of these is a fruit?" The options are: A) Car, B) Dog, C) Apple, D) Truck.
You don't even need to read the question to know the answer is Apple. It's the only fruit. The AI can guess the answer just by looking at the choices, without understanding the question. This is a "shortcut."
The BenchMarker Fix: BenchMarker plays a game of "Guess the Question." It takes the choices (Car, Dog, Apple, Truck) and asks a super-smart AI, "What question do you think goes with these answers?" If the AI can guess the question just by looking at the choices, it means the exam has a shortcut.

The Result: They found that some exams are so full of these tricks that the AI is essentially playing a guessing game, not a thinking game.

3. The "Grammar Police" Check (Writing Errors)

The Problem: Imagine a test where some questions are written by a native English speaker, but others are written by a robot that forgot to put a period at the end, or where the answer choices don't match the grammar of the question (e.g., the question asks for a plural noun, but all the answers are singular). This confuses the student and makes the test unfair.
The BenchMarker Fix: BenchMarker uses a 19-point checklist borrowed from real education experts. It checks for things like:

Is the grammar consistent?
Are the answers too long or too short compared to each other?
Is there only one correct answer?
The Result: They found that some AI benchmarks are a mess. For example, in one dataset called HellaSwag, every single question broke at least two of these grammar rules.

Why Does This Matter?

The paper shows that when you fix these broken questions, the scores change dramatically.

The "Fake" High Scores: When the AI takes a test full of "cheating" questions (contamination), it gets a high score. But if you remove those questions, the score drops. The AI wasn't actually that smart.
The "Fake" Low Scores: When the test is full of typos and bad grammar, the AI gets a low score. But that doesn't mean the AI is dumb; it just means the test was poorly written.
The Ranking Shuffle: The most shocking finding is that fixing these errors changes the leaderboard. The AI model that was ranked #1 might drop to #3 once you clean up the test. It's like a race where the winner crossed the finish line early because the track was broken.

The Big Lesson

The authors say that for a long time, AI researchers have been trying to fix these tests by just rewriting them. But they often fix one problem and accidentally create a new one (like making the questions so hard that they become confusing).

BenchMarker is the solution. It's a toolkit that helps researchers audit their exams, just like a teacher reviews a test before handing it out. By borrowing rules from human education, they hope to build better, fairer, and more honest tests for Artificial Intelligence.

In short: BenchMarker is the tool that stops AI from cheating, guessing, and getting confused by bad grammar, ensuring that when we say an AI is "smart," we actually mean it.

1. Problem Statement

Multiple-Choice Question Answering (MCQA) is the standard evaluation paradigm for Natural Language Processing (NLP) models. However, existing benchmarks suffer from a lack of rigorous quality control, leading to "evaluation crises" where model performance does not reflect true capability. The paper identifies three primary categories of flaws that undermine the construct validity of these benchmarks:

Contamination: MCQs that appear verbatim or nearly verbatim in the pre-training data of Large Language Models (LLMs), allowing models to "cheat" via memorization rather than reasoning.
Shortcuts: Structural cues within the answer choices that allow models to guess the correct answer without reading the question stem (e.g., the "odd one out" strategy or grammatical inconsistencies).
Writing Errors: Structural, semantic, and grammatical issues (e.g., ambiguous stems, implausible distractors, inconsistent grammar) that make items misleading or unanswerable, a problem long recognized in educational assessment but often ignored in NLP.

Prior work has attempted to fix specific benchmarks, but lacks reusable, automated metrics to verify if fixes actually reduce errors or inadvertently introduce new ones.

2. Methodology: The BenchMarker Toolkit

The authors propose BenchMarker, a toolkit inspired by decades of educational research (specifically the 19-rule Item-Writing Flaws rubric by Tarrant et al.) that utilizes LLM-as-a-Judge to automate the detection of these flaws.

Core Components

BenchMarker evaluates MCQs across three axes using specific LLM prompts and search APIs:

Contamination Detection:
- Method: Uses a web search API (e.g., Google) to query the question stem and gold answer.
- Judgment: An LLM judge analyzes search results to determine if the MCQ exists exactly or nearly exactly online. If found, it is flagged as contaminated (proxy for pre-training data leakage).
Shortcuts Detection:
- Method: Employs a "Choices-Only" strategy.
- Process:
  1. Three high-performing LLMs (GPT-5, Gemini 2.5 Pro, Claude 4.5 Sonnet) attempt to answer the MCQ using only the choices (no question stem).
  2. These models are prompted to "guess" the original question stem based on their reasoning.
  3. A judge LLM compares the inferred question to the original question.
- Flagging: If the LLMs can solve the item via choices alone and the inferred question does not semantically match the original, the item is flagged as having a problematic shortcut.
Writing Error Detection:
- Method: Applies a 19-rule rubric derived from educational standards (e.g., "Grammatical Consistency," "Plausible Distractors," "No Absolute Terms").
- Process: For each of the 19 rules, a specific LLM judge is prompted with the rule definition, examples, and the MCQ to determine if a violation exists ("pass" or "fail").
- Aggregation: An item is considered flawed if it violates one or more rules.

Validation

The authors validated BenchMarker against 8,042 human judgments across 12 diverse MCQA datasets (including MMLU, TruthfulQA, HellaSwag, and student exams).

Human Agreement: The best LLM judges (GPT-5, Gemini Pro) achieved high Cohen's $\kappa$ (0.61–0.66) with human experts, significantly outperforming trivial baselines and existing tools like SAQUET.
Generalization: The tool was tested on both NLP-generated datasets and human-written educational exams, demonstrating robustness across domains.

3. Key Results

The authors audited 12 major benchmarks using BenchMarker, revealing pervasive issues:

Prevalence of Flaws:
- Contamination: 47% of TruthfulQA items appear online; HellaSwag had 0% contamination (likely due to age), but MMLU had 45%.
- Shortcuts: 23% of SocialIQA items contain exploitable shortcuts.
- Writing Errors: HellaSwag violated at least two writing rules in 100% of its items. On average, crowdsourced and auto-generated datasets had significantly higher error rates than educator-written exams.
Impact on Evaluation:
- Accuracy Inflation: Contaminated splits showed higher accuracy (models memorized answers).
- Accuracy Deflation: Splits with writing errors showed lower accuracy (models were confused by poor phrasing).
- Ranking Shifts: Removing writing errors shifted LLM rankings beyond random chance (up to 2 positions), meaning flawed benchmarks lead researchers to select suboptimal models for training and deployment.
The "Fix" Paradox:
- Revised versions of benchmarks (e.g., MMLU-Pro, GoldenSwag) successfully addressed their target issues (e.g., lowering accuracy via harder distractors) but inadvertently introduced new flaws.
- MMLU-Pro introduced implausible distractors and multiple correct answers.
- GoldenSwag improved grammar but increased grammatical inconsistencies due to automated filtering.
- Simple LLM prompting to "rewrite" flawed questions failed to eliminate errors and often created new ones.

4. Key Contributions

BenchMarker Toolkit: A reusable, open-source toolkit (integrated with InspectAI) that automates the detection of contamination, shortcuts, and 19 types of writing errors using LLM judges.
Comprehensive Audit: A rigorous analysis of 12 NLP benchmarks demonstrating that flaws are ubiquitous, particularly in crowdsourced and auto-generated data, and that they significantly distort model rankings.
Educational-NLP Bridge: The first systematic application of established educational assessment rubrics (19-item rule set) to NLP benchmark auditing, validating that education research offers a path forward for NLP evaluation.
Validation Dataset: A new dataset of 8,042 human-labeled MCQs for testing LLM judge reliability in flaw detection.
Critical Analysis of Fixes: Evidence that "fixing" benchmarks is a multi-objective problem where solving one issue often creates another, necessitating iterative refinement rather than one-off corrections.

5. Significance and Future Work

Evaluation Rigor: The paper argues that NLP is facing an "evaluation crisis" where poor benchmark quality erodes trust in model progress. BenchMarker provides a mechanism to restore rigor by treating benchmarks with the same quality control standards as human educational assessments.
Paradigm Shift: It challenges the NLP community to move beyond simple accuracy metrics and consider the construct validity of the questions themselves.
Future Directions: The authors suggest using BenchMarker as a "verifier" in iterative rewriting loops, integrating it into user interfaces for human-in-the-loop editing, and expanding the toolkit to cover non-English languages and specialized domains (e.g., medicine, coding).

In summary, BenchMarker demonstrates that the quality of the questions is as critical as the quality of the models. By adopting educational standards, the NLP community can build more reliable, fair, and valid benchmarks for the future of AI evaluation.