💬 NLP

Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options

This paper introduces a massive option evaluation protocol with up to 100 choices to expose how conventional low-option benchmarks overstate large language model competence by masking true abilities through shortcut strategies and positional biases.

Original authors: Nahyun Lee, Guijin Son

Published 2026-04-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nahyun Lee, Guijin Son

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a multiple-choice test.

The Old Way (The "Easy" Test):
Usually, these tests have 4 or 5 options. If you get 95% or 100% correct, we assume you are a genius. But here's the catch: sometimes, you don't actually know the answer. You might just be guessing that "C" is usually the right answer, or you might spot a tiny formatting clue that gives it away. It's like playing a game of "Hide and Seek" where the person hiding is only behind one of four curtains. Even if you aren't very good at finding things, you'll probably find them just by luck.

The New Way (The "Massive" Test):
The authors of this paper asked: "What if we made the game harder?" Instead of 4 curtains, what if there were 100 curtains?

If you are truly smart, you can still find the right person behind the correct curtain, even if there are 99 other people hiding there who look almost exactly the same. But if you were just guessing or using "tricks" to win the easy game, you will suddenly crash and burn when faced with 100 options.

The Experiment: The Korean Spelling Challenge

The researchers created a specific test to try this out. They asked AI models to find one single sentence with a spelling mistake hidden among 99 perfectly correct sentences.

Think of it like this: Imagine a room full of 100 identical twins. One of them has a tiny, almost invisible smudge of dirt on their cheek. Your job is to point to the dirty twin.

The "Easy" version: There are only 4 twins. It's easy to spot the dirt.
The "Hard" version: There are 100 twins. The dirt is still there, but now the "noise" of 99 clean twins is overwhelming.

What They Found

They tested several famous AI models (like Gemini, HyperCLOVAX, and EXAONE). Here is what happened:

The "Bubble" Burst:
On the easy test (4 options), many models looked like super-geniuses, getting nearly 100% correct. But when the test jumped to 100 options, some of these "geniuses" suddenly dropped to failing grades.
- Analogy: It's like a student who aces a quiz with 4 questions but fails a final exam with 100 questions because they were just memorizing patterns, not actually learning the material. The "bubble" of their high scores popped.
The "First Option" Panic:
When the AI models got confused by the 100 options, they didn't just guess randomly. They started panicking and falling back on a lazy habit: picking the first few options.
- Analogy: Imagine you are looking for a specific book in a library with 100,000 books. If you can't find it, you might just grab the first book on the shelf and hope for the best. The AI started doing this. It stopped reading the content and started guessing based on where the answer was, rather than what the answer was.
It Wasn't About "Memory" (Context Length):
A common worry is that AI gets confused because the list of 100 options is too long to "read" all at once (like a short-term memory limit).
- The Fix: The researchers added "filler text" (nonsense words) to make the list long even when there were only 4 options.
- The Result: The AI didn't get confused by the length. It only got confused when there were too many similar choices. The problem wasn't that the list was long; the problem was that the choices were too hard to tell apart.

Why Does This Matter?

This paper is a wake-up call for the AI world.

Current Benchmarks are "Too Easy": Just because an AI gets 99% on a standard test doesn't mean it's truly smart. It might just be good at playing the "4-option game."
Real Life is Hard: In the real world, we don't have 4 options. We have infinite possibilities. If an AI can't distinguish the right answer from 99 very similar wrong answers, it's not reliable enough for real-world use.
The Solution: We need to stress-test AI with "massive option" challenges. This forces the AI to prove it actually understands the content, rather than just guessing or using shortcuts.

In short: The paper says, "Stop letting AI pass by guessing. Give them a test with 100 options to see if they really know what they are doing."

1. Problem Statement

Current Large Language Model (LLM) benchmarks rely heavily on Multiple Choice Question Answering (MCQA) with a small number of options ( $N \in \{4, 5\}$ ). The authors argue that this setup suffers from a fundamental structural limitation:

Ceiling Effects & Shortcut Strategies: As models approach near-perfect accuracy in low- $N$ settings, high scores often reflect "shortcut strategies" (e.g., partial elimination, distributional guessing, or exploiting formatting cues) rather than genuine reasoning or knowledge.
Obscured Competence Gaps: These shortcuts allow models to mask their true limitations, making it difficult to distinguish between models that truly understand the content and those that are merely navigating a limited search space.
Need for Stress Testing: There is a lack of evaluation protocols that can test model reliability under "dense interference," where the model must discriminate a correct answer from a large pool of highly confusable distractors.

2. Methodology

The authors propose a Massive-Option Evaluation Protocol ( $N=100$ ) applied to a specific linguistic task to stress-test model robustness.

Task Formulation: Korean Orthography Error Detection

Objective: The model must identify the single orthographically incorrect sentence from a pool of $N$ candidates (where $N-1$ are correct).
Rationale: Orthography offers objective verifiability and high surface similarity between the target error and correct distractors, forcing the model to rely on exact knowledge retrieval rather than semantic interpretation.
Dataset Construction:
- Source: 750 Korean sentences from the National Institute of Korean Language (NIKL).
- Labeling: A strict consensus protocol using three spell-checkers (DAUM, SARAMIN, NARA). A sentence is a valid distractor only if all checkers agree it is correct; a target is valid only if all agree it is erroneous.
- Difficulty Levels: Sentences are scored (Levels 1–4) based on morpho-syntactic features (e.g., clause endings, dependent nouns).
- Environments: Two distractor pools were created: Easy (Levels 1–2) and Full (Levels 1–4).

Experimental Protocol

Option Scaling: Models are tested at $N \in \{4, 10, 20, 50, 100\}$ .
Dynamic Sampling: For each of the 30 fixed target errors, $K=1,000$ independent trials are run. In each trial, $N-1$ distractors are randomly sampled from the environment pool, and the target's position is shuffled.
Models Evaluated: Frontier multilingual and Korean-specialized LLMs (Gemini 2.5/3, A.X-4.0, EXAONE-4.0, HyperCLOVAX-THINK).

Key Metrics & Diagnostics

Chance-Normalized Accuracy (NA): Adjusts raw accuracy to account for the random guess baseline ( $1/N$ ), scaling performance from 0 (random) to 1 (perfect).
$NA(N) = \frac{A(N) - 1/N}{1 - 1/N}$
Bubble Index (BI): Quantifies the "competence inflation" of low- $N$ settings relative to $N=100$ . A high BI indicates that low- $N$ accuracy is misleading.
$BI = 1 - \frac{NA(100)}{NA(4) + \epsilon}$
Positional Diagnostics:
- Excess Primacy ( $\Delta PFI_{10}$ ): Measures if models disproportionately select the first 10 options compared to the uniform distribution of the correct answer.
- Gold-Position Slope: Determines if accuracy drops as the correct answer appears later in the list (indicating a "Lost in the Middle" or early-option bias).
Padding Controls: To isolate whether performance drops are due to context length or ranking complexity, the authors injected task-irrelevant padding (English text, symbols, Korean prose) to match the token count of $N=100$ prompts without changing the option count.

3. Key Results

Performance Collapse at High $N$

Gemini Models: Maintained high robustness. In the Full environment, Gemini-3-Pro-preview dropped from ~98% (Acc4) to ~85% (Acc100), but its chance-normalized accuracy remained high, showing true competence.
Other Models (HyperCLOVAX, EXAONE): Exhibited catastrophic degradation.
- HyperCLOVAX-THINK: Dropped from 92.5% (Acc4) to 21.2% (Acc100) in the Full environment.
- EXAONE-4.0: Dropped from 81.2% (Acc4) to 14.0% (Acc100).
Bubble Index: Models with large BIs (e.g., EXAONE: 0.82) revealed that their low- $N$ success was largely an artifact of the small search space.

Failure Modes Identified

Semantic Confusion: In the Full environment, models struggled to distinguish the target from highly similar distractors, leading to random guessing.
Positional Fallback (The "Early Option" Bias): Under uncertainty, weaker models abandoned content analysis and defaulted to selecting the first few options.
- HyperCLOVAX-THINK showed a massive Excess Primacy ( $\Delta PFI_{10} = 0.526$ ), meaning over 50% of its "guesses" were concentrated in the first 10 options, far exceeding the uniform baseline.
- This behavior correlated with a negative gold-position slope, confirming that accuracy was driven by position, not content.

Context Length vs. Ranking Complexity

Padding Experiments: When padding was added to low- $N$ tasks to match the token length of $N=100$ tasks, performance remained stable for most models (e.g., Gemini, A.X-4.0).
Conclusion: The performance collapse at $N=100$ is not primarily caused by long-context limitations (token count) but by the difficulty of candidate ranking under dense interference.
Exception: HyperCLOVAX-THINK showed sensitivity to English-text padding, suggesting a specific vulnerability to distribution shifts rather than a generic length limit.

4. Key Contributions

Massive-Option Framework: Introduced a scalable evaluation protocol ( $N=100$ ) that effectively eliminates chance performance and exposes "competence bubbles" hidden in standard 4-option benchmarks.
Diagnostic Tools: Developed quantitative metrics (Bubble Index, Excess Primacy, Gold-Position Slope) to distinguish between genuine reasoning failures and heuristic fallback strategies.
Empirical Evidence of Failure Modes: Demonstrated that models with near-perfect low- $N$ scores can suffer >70% accuracy drops at high $N$ , often reverting to primitive positional heuristics rather than engaging in content analysis.
Disentanglement of Factors: Proved via padding controls that the primary bottleneck in massive-option evaluation is ranking complexity, not context length.

5. Significance and Implications

Rethinking Benchmarks: The paper argues that next-generation benchmarks must move beyond broad subject coverage to include "stress tests" involving dense, confusable alternatives.
Reliability in Deployment: Real-world applications often involve searching through large, noisy candidate sets. Low- $N$ benchmarks fail to predict model behavior in these high-interference regimes.
Model Selection: The study reveals that models appearing equal on standard benchmarks (e.g., MMLU, HAE-RAE) have vastly different robustness profiles. Models like Gemini demonstrate superior reliability under interference, while others rely on shortcuts that fail when the search space expands.
Future Directions: The authors call for evaluation settings that explicitly probe decision reliability and semantic interference, ensuring models can deterministically identify correct answers even when distractors are highly confusable.