💬 NLP

Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

This paper introduces the LLM Nominal Response Model (LLM-NRM), a psychometric framework that leverages the full distribution of answer choices—including incorrect options—to more accurately estimate LLM abilities and item characteristics, demonstrating that distractor preferences provide significant measurement value and enable highly efficient benchmarking compared to traditional binary scoring.

Original authors: Xiao Fei, Yang Zhang, Sarah Almeida Carneiro, Michalis Vazirgiannis

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiao Fei, Yang Zhang, Sarah Almeida Carneiro, Michalis Vazirgiannis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a multiple-choice test. In the traditional way of doing things, you only care about the final mark: a checkmark for right, a cross for wrong. If a student gets a question wrong, you just see a red "X." You don't know why they got it wrong. Did they guess randomly? Did they know a little bit but get confused by a tricky word? Or did they just hate the color of the letter "B" and pick that one? For a long time, scientists studying Artificial Intelligence (AI) have done the exact same thing. They ask AI models multiple-choice questions and only count the correct answers. This is like judging a chef only by whether the dish is edible, ignoring whether they used too much salt, forgot the pepper, or accidentally served soup with a fork.

This paper steps into the world of psychometrics—the science of measuring human (and now machine) abilities. It builds on a clever idea called Item Response Theory, which is basically a fancy way of saying that a test question isn't just "hard" or "easy"; it has a personality. Some questions trick smart people, while others trick beginners. The big question this paper asks is: If we stop treating every wrong answer as a simple failure, can we learn something new? The authors suggest that the specific wrong answer an AI picks is actually a treasure trove of information, revealing how the AI thinks, what it confuses, and how confident it feels, even when it's wrong.

The Paper's Big Idea: Why Wrong Answers Are Actually Right

The researchers, led by Xiao Fei and colleagues, introduced a new tool called LLM-NRM (LLM Nominal Response Model). Think of this as upgrading the grading system from a simple "Pass/Fail" sheet to a detailed "Forensic Report."

In the old way, if an AI is asked, "Which object in the solar system is orbited by a belt of asteroids?" and it picks "Saturn" instead of the correct "the Sun," the system just records a mistake. But the new model looks closer. It realizes that picking "Saturn" is different from picking "Pluto" or "the Moon." Maybe the AI knows Saturn has rings and got confused with asteroid belts. Maybe it just likes the sound of the word "Saturn." By analyzing which wrong answer the AI chose, the model can map out the AI's "brain" much more accurately than just counting correct answers ever could.

The team tested this on a massive scale: 189 different AI models and 31,554 questions from 14 different benchmarks. They found that their new method was a huge improvement. When they tried to predict how an AI would answer a question it hadn't seen before, their model was much more accurate than the old binary (right/wrong) methods.

The Secret Sauce: What Makes LLM-NRM Special?

The paper argues that AI models aren't just "smart" or "dumb"; they have specific quirks that the old tests ignore. The new model accounts for three weird behaviors that happen when AI takes a test:

Confidence vs. Knowledge (Calibration Sharpness): Sometimes an AI is very sure it's right, even when it's wrong. Other times, it's unsure even when it's right. The old tests couldn't tell the difference. The new model has a "sharpness" dial that measures how confidently the AI spreads its bets across the answers.
The "B" Bias (Positional Preference): Humans and AIs sometimes have a weird habit of liking the first or last answer choice just because of where it sits on the page, not because of what it says. The new model spots this bias and subtracts it out, so it doesn't look like the AI is smarter than it really is.
The "Guessing" Fallback: When a question is too hard, the AI might just give up and pick an answer based on a random rule (like "always pick the longest word"). The new model detects when the AI is doing this "fallback" behavior and separates it from its actual knowledge.

The Big Reveal: Wrong Answers Are Gold

The most exciting finding is that wrong answers carry more information than you'd think. The authors calculated that looking at which wrong answer was chosen adds about 101% more useful information per question than just knowing if it was right or wrong.

To prove this, they did a cool experiment. They tried to guess an AI's overall skill level using only the questions the AI got wrong. Surprisingly, just by looking at the pattern of mistakes, they could estimate the AI's ability with a correlation of 0.943 (a very high score, where 1.0 is perfect). This means you don't even need to see the correct answers to understand how smart the AI is; the mistakes tell the whole story.

Making Tests Smarter and Faster

Because this new model gets so much information out of every single question, it can make testing way more efficient.

Fewer Questions Needed: Usually, to rank AI models accurately, you need hundreds of questions. The authors found that with their new method, you could pick just 41 of the most informative questions and still get a ranking that matches the full test almost perfectly. That's a 770 times reduction in the number of questions needed!
Fewer Models Needed: Usually, you need thousands of AI models to "calibrate" a test (figure out how hard the questions are). This new method can calibrate a test using as few as 5 AI models and still get good results, whereas older methods get messy and unreliable with so few models.

What This Means for the Future

The paper concludes that we have been throwing away a massive amount of data by treating every wrong answer as the same. By listening to how an AI gets things wrong, we get a clearer, fairer, and faster picture of what it can actually do. It's like realizing that a student's wrong answer isn't just a failure, but a map showing exactly where their understanding stops and where their confusion begins. The authors suggest that this approach could help us build better tests and understand AI behavior much deeper than just a simple score on a report card.

Technical Summary: Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

Problem Statement

Current Large Language Model (LLM) benchmarking relies heavily on binary accuracy scores derived from Multiple-Choice Questions (MCQs). This approach treats all incorrect responses as equivalent, discarding the systematic information contained in which specific distractor an LLM selects. Consequently, accuracy scores are limited by their dependence on specific item compositions and fail to capture model confidence, systematic error patterns, or measurement uncertainty. As models approach benchmark saturation, binary scoring loses discriminative power. Furthermore, existing psychometric applications to LLMs (e.g., Item Response Theory) typically reduce responses to binary correctness, failing to leverage the full distribution of option preferences that LLMs naturally produce.

Methodology: The LLM Nominal Response Model (LLM-NRM)

The authors propose the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that adapts Bock's Nominal Response Model (NRM) for LLM evaluation. Unlike binary models, LLM-NRM models the complete probability distribution over all answer choices to jointly estimate latent LLM ability ( $\theta$ ) and item characteristics.

Core Formulation

The model treats an LLM's response to an MCQ item as a distribution $p_{ji}$ over $K_i$ options, rather than a single binary label. It extends the classical NRM utility function $u_{jik} = a_{ik}\theta_j + c_{ik}$ by incorporating three LLM-specific behavioral factors:

Response Calibration Sharpness ( $s_j$ ): A per-LLM parameter acting as an inverse temperature. It controls the concentration of the probability distribution without altering the relative ordering of options, capturing whether a model expresses preferences with high confidence or is "flat" in its distribution, independent of item knowledge.
Positional Bias ( $\delta_j$ ): A vector representing content-independent preferences for specific answer positions (e.g., A, B, C, D). This separates systematic positional heuristics from the actual attractiveness of the option content.
Difficulty-Gated Guessing Fallback: A mechanism that interpolates between ability-driven preferences and a fallback strategy when an item is too difficult for the model.
- A gate $G_{ji} = \sigma[w_0 - \kappa(\theta_j - \tilde{b}_i)]$ determines the weight of the fallback, where $\tilde{b}_i$ is an approximated item difficulty index derived from NRM parameters.
- A fallback logit vector $\rho_j$ defines the specific distribution of guesses (e.g., label priors) when the gate is active.

The final response distribution is a softmax interpolation:
$p_{jik}^{LLM-NRM} = \frac{\exp[(1-G_{ji})u_{jik} + G_{ji}\rho_{jk}]}{\sum_{k'} \exp[(1-G_{ji})u_{jik'} + G_{ji}\rho_{jk'}]}$

Estimation

Parameters are estimated via Maximum A Posteriori (MAP) using Adam optimization on a soft cross-entropy loss, regularized by a standard normal prior on ability ( $\theta_j \sim \mathcal{N}(0,1)$ ). The model is fitted on a dataset of 189 LLMs and 31,554 MCQ items across 14 benchmarks.

Key Contributions

First Application of NRM to LLMs: This work introduces the Nominal Response Model to the LLM domain, revealing latent response preferences across options and providing a richer view of model behavior beyond correctness.
LLM-Adapted Formulation: The authors propose a novel formulation that explicitly models calibration sharpness, positional bias, and difficulty-dependent fallback behaviors, moving beyond traditional item-response modeling to capture specific LLM decision processes.
Demonstration of Distractor Utility: The paper establishes that incorrect responses are not random noise but contain distinct, useful measurement information.

Experimental Results

Predictive Performance

Evaluated on held-out data via 5-fold cross-validation, LLM-NRM outperforms binary IRT baselines (1PL–4PL), standard NRM, and prior state-of-the-art approaches (e.g., Deep-IRT, PSN-IRT).

Binary Correctness: LLM-NRM achieves the highest accuracy (0.8395) compared to the best binary baseline (0.8232).
Option-Level Prediction: The gains are more pronounced in predicting the full option distribution, where LLM-NRM achieves 0.7070 accuracy and 0.8181 log-loss, significantly outperforming binary baselines (0.6427 accuracy, 0.9879 log-loss) and standard NRM.

External Validity

The ability estimates ( $\theta$ ) derived from LLM-NRM show the strongest Spearman correlation (0.920) with the external Arena.ai Text Elo leaderboard (based on human preferences), outperforming raw accuracy (0.866) and other latent ability estimates (max 0.885). This suggests that option-level patterns capture model differences aligned with human judgment better than aggregate correctness.

Information Content of Wrong Answers

Fisher Information: The identity of the selected distractor contributes an additional +101% Fisher Information per item on average compared to binary correctness alone.
Recovery from Errors: Even when observing only incorrect responses (ignoring correctness signals), the model can recover ability estimates with a Spearman correlation of 0.9428 against the full-information reference. This confirms that error structures are highly informative and systematic.

Benchmark Efficiency

Item Compression: Using LLM-NRM to select the 41 most informative items allows for a 770x reduction in benchmark size while preserving the full-bank ranking with a Kendall correlation of 0.85.
Calibration Efficiency: LLM-NRM can calibrate item parameters using as few as 5 LLMs and approaches full-fleet performance with ~40 items, significantly outperforming binary IRT models which require larger fleets to stabilize due to the limited information in binary responses.

Significance and Claims

The paper concludes that incorrect answers in LLM benchmarks carry distinct and useful measurement information rather than representing equivalent mistakes. By modeling the full distribution of responses, LLM-NRM:

Improves Evaluation Accuracy: Provides more reliable ability estimates that align with human preferences.
Reveals Behavioral Nuances: Separates true knowledge from calibration artifacts, positional biases, and fallback strategies.
Enables Efficiency: Drastically reduces the cost of benchmarking by enabling high-fidelity ranking with fewer items and fewer calibration models.

The authors assert that this approach transforms the evaluation paradigm from a binary "right/wrong" metric to a nuanced psychometric analysis where "every wrong answer counts." Future work is suggested to explore partial probability observations, multidimensional abilities, and extensions beyond fixed-option questions, but the current work focuses on validating the utility of option-level data in existing MCQ frameworks.