💬 NLP

The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

This paper reveals a "benchmark illusion" where pruned large language models appear competent on multiple-choice evaluations but fail in open-ended generation because pruning demotes correct answers rather than erasing them, suggesting that compressed models must be tested on their generative capabilities rather than just their recognition abilities.

Original authors: Rui Wen, Lu Sun, Jiayang Liu, Zesheng Xu, Tianshuo Cong, Zheng Li

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rui Wen, Lu Sun, Jiayang Liu, Zesheng Xu, Tianshuo Cong, Zheng Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian (a Large Language Model) who knows the answer to almost any question. To save money and space, you decide to "prune" the library. You remove a huge chunk of the books and shelves, keeping only the most essential ones.

The paper you're asking about investigates what happens when you test this shrunken library.

The "Multiple-Choice Illusion"

Here is the trick: When you ask the librarian a question, you can test them in two ways:

The Open Question: You ask, "Who discovered the Azores?" and wait for them to speak the answer from memory.
The Multiple-Choice Test: You ask, "Who discovered the Azores? A) 1427, B) 1500, C) 1600, D) 1700."

The paper found a surprising "illusion." After pruning, the librarian often fails the Open Question. They might say, "I think it was the 15th century," which is vague or wrong. But, if you give them the Multiple-Choice test, they ace it. They immediately point to "1427" and get it right.

Standard benchmarks (the tests used to rate AI) usually rely on the Multiple-Choice format. This creates a false sense of security. The test says, "Your library is still perfect!" but in reality, the librarian has lost the ability to recall the answer without help. They can still recognize the answer if you show it to them, but they can't produce it on their own.

The "Demotion" vs. "Erasure" Theory

The authors asked a critical question: Did pruning erase the knowledge (the librarian forgot the fact entirely), or did it just demote the knowledge (the librarian still knows it, but it's no longer their first guess)?

They discovered it's mostly demotion.

Think of the librarian's mind like a crowded room of ideas. Before pruning, the correct answer was standing right at the front door, waving at you. After pruning, the correct answer is still in the room, but it's been pushed back to the third or fourth row.

Greedy Decoding (The default setting): The librarian only looks at the person standing at the front door. Since the correct answer is now in row 3, the librarian picks the wrong person standing at the front.
Multiple Choice: You hand the librarian a list of people in the room. Even though the right person is in row 3, the librarian can still see them on the list and pick them.

How to Fix the "Demotion"

Because the knowledge wasn't erased, just pushed back, the paper shows you can often get the answer back with a little help:

Wider Search: Instead of just looking at the front door (greedy decoding), if you tell the librarian to look at the top 5 people in the room (Beam Search) or take a few guesses (Sampling), they will find the correct answer again.
A Tiny Hint: If you give the librarian one example of how to answer a similar question (one "in-context" example), it helps them push the correct answer back to the front of the line.

The Catch

This "demotion" effect happens a lot, especially with larger models and languages the model is good at. However, the paper notes that for smaller models or very difficult languages, the pruning sometimes actually erases the knowledge completely. In those cases, even if you show the librarian the list, they can't find the answer because it's gone.

The Bottom Line

The paper warns us not to trust "Multiple-Choice" scores too much when judging compressed AI models. A model might look perfect on a test where you give it the answers, but fail miserably when a real user asks a direct question. To know if a compressed model is truly useful, we need to test what it can produce on its own, not just what it can recognize when we hold the answer key up to it.

Technical Summary: The Benchmark Illusion in Pruned LLMs

Problem Statement

Large Language Models (LLMs) are frequently compressed via pruning to reduce memory usage and inference costs. Developers typically rely on standard multiple-choice (MC) benchmarks to validate that model performance remains intact post-compression. However, this paper identifies a critical "benchmark illusion": a pruned model may achieve high scores on MC evaluations while failing to generate the correct answer in open-ended, greedy generation tasks.

The core question investigated is whether pruning erases knowledge entirely or merely demotes the correct answer, making it difficult to produce as the top output. The authors argue that standard MC benchmarks create an evaluation blind spot by overestimating the usability of compressed models, as they test recognition rather than production.

Methodology

The study employs a paired-item diagnostic framework to distinguish between "recognizing" an answer and "producing" it. The methodology involves:

Stress Test Setup: The authors utilize multilingual Question Answering (QA) using the TyDiQA and XQuAD datasets. They select questions where the uncompressed (dense) model answers correctly in both open generation and MC formats.
Compression: Models (including Qwen3-8B, Mistral-7B-Instruct, and Phi-3-mini) are pruned using one-shot methods, specifically WANDA (high-sparsity) and SparseGPT, without retraining.
Three-Level Evaluation: The same questions are evaluated across three formats to measure the "Generation Gap" ( $\Delta_{PR}$ $Δ_{P R}$ ):
- Spontaneous Generation: Greedy decoding without candidates (hardest).
- Prompted Recognition (Candidate-Shown): The model is given the correct answer and distractors and must generate the answer.
- Multiple-Choice (MC) Scoring: The model scores options via log-likelihood to select the best one (easiest).
Diagnostic Metrics:
- Dissociation Rate: The fraction of items that were correct in both formats pre-pruning but become "MC-only" (correct in MC, wrong in generation) post-pruning.
- Token Rank Analysis: Inspecting the rank of the gold answer's first token in the probability distribution to determine if it is erased or merely demoted.
- Recoverability: Testing if broader decoding methods (Beam Search, Nucleus Sampling) or light prompting (1-shot in-context examples) can recover the correct answer from pruned models.

Key Findings

1. The "MC-Only" Phenomenon

Pruning frequently breaks greedy open generation while leaving MC recognition intact.

Dissociation: In high-sparsity pruning (e.g., WANDA at 0.5 sparsity), a significant portion of items (ranging from 12% in English to 50% in Bengali) become "MC-only" errors. The model fails to generate the answer but successfully selects it from a list.
Support Gradient: Accuracy drops are minimized as the evaluation format provides more answer support. Open generation suffers the largest drop (e.g., 8.7 percentage points), while MC scoring suffers the least (2.7 percentage points).

2. Demotion, Not Erasure

The paper demonstrates that pruning typically demotes the correct answer rather than erasing it.

Token Ranking: For items that become "MC-only" after pruning, the gold answer's first token retains a median rank of 3.5 in the probability distribution. This is a "near miss" (close to the top) rather than a deep burial (e.g., rank 17 or higher, seen in items the dense model already failed).
Mechanism: Pruning shifts probability mass near the top of the distribution. The correct token often drops from rank 1 to rank 3 or 4, causing greedy decoding to select a competitor. Once the first token is wrong, errors compound.

3. Recoverability

Because the knowledge is demoted rather than erased, it is often recoverable:

Broader Decoding: Using Beam Search (beam size 5) or Nucleus Sampling (10 samples) significantly recovers accuracy for pruned models. For example, Swahili accuracy rises from 39% (greedy) to 74% (10-sample).
Stability: While greedy accuracy drops by ~9.1 percentage points after pruning, the "oracle recoverability" (whether the answer exists in the distribution) drops by only ~0.7 percentage points.
Prompting: Adding a single in-context example also helps reduce the performance gap between dense and pruned models.

4. Limitations and Boundaries

The "demotion" hypothesis does not hold universally.

Model Size: Smaller Qwen checkpoints are more likely to lose the underlying recognition ability itself, rather than just demoting the answer.
Language/Tokenization: Recovery rates vary by language. Low-resource languages with complex tokenization (e.g., Bengali, Telugu) show lower recovery rates (12%) compared to others (e.g., Swahili at 59%), suggesting genuine knowledge loss in these specific cases.

Contributions

A New Diagnostic Tool: Introduction of a paired-item test comparing generation, recognition, and reachability to expose compression-induced failures.
Exposing the Benchmark Illusion: Demonstration that aggressive pruning often destroys greedy generation capabilities while preserving MC recognition, leading to inflated benchmark scores.
Mechanistic Explanation: Evidence that pruning primarily demotes correct answers to lower ranks (near misses) rather than deleting them, making them recoverable via broader decoding or light prompting.

Significance

The paper concludes that retained benchmark scores do not guarantee retained usability. Standard MC benchmarks create a blind spot by testing what a model can recognize rather than what it can produce. To accurately assess the real-world utility of compressed LLMs, evaluation must prioritize open-ended generation and test the model's ability to produce answers, not just select them from a list. The "benchmark illusion" suggests that current compression techniques may be more fragile in practical deployment than leaderboards indicate.