💬 NLP

Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers

This paper challenges the assumption that large language models' success on multiple-choice questions using only answer choices is always a shallow shortcut, demonstrating that test-time reasoning often improves performance through more robust strategies like inferring missing questions rather than relying on trivial data biases.

Original authors: Nishant Balepur, Atrey Desai, Rachel Rudinger

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nishant Balepur, Atrey Desai, Rachel Rudinger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a multiple-choice test, but someone has ripped the question out of the paper. You are left with just the four possible answers (A, B, C, and D).

In the past, researchers thought that if a computer (a Large Language Model, or LLM) could still guess the right answer without seeing the question, it was "cheating." They assumed the computer was just spotting silly patterns, like "Answer C is the only one with a comma," or "Answer B is the longest word." They thought the computer wasn't actually smart; it was just a trickster.

This paper says: "Hold on. Let's look at the computer's scratch paper."

The authors gave 12 different advanced AI models a bunch of these "question-less" tests. But here's the twist: they asked the AIs to think out loud before answering. They forced the models to write down their reasoning step-by-step, like a student showing their work on a math test.

Here is what they found, explained simply:

1. The "Cheater" Might Actually Be a Detective

When the AI looked at just the answers, it was surprisingly good at guessing the right one.

The Old View: "It's just guessing based on word length or weird formatting. It's a cheat."
The New View: The AI's "scratch paper" (the reasoning trace) showed it was doing real detective work.
- The "Odd One Out" Strategy: Sometimes the AI noticed that three answers were about "kitchen items" and one was a "car engine." Even without the question, it guessed the question was likely about kitchen items, so it picked the car engine as the outlier.
- The "Fact Checker" Strategy: The AI would read the answers and say, "Wait, spiders don't eat grass. That answer is biologically impossible. I'll cross it out."
- The "Question Reconstructor" Strategy: The AI would look at the answers and say, "These answers are all about renewable energy. The missing question must be asking about renewable resources." Then it would solve the problem based on that guess.

The Analogy: Imagine you walk into a room and see four people: three are wearing chef hats, and one is wearing a firefighter helmet. You don't know what the room is for, but you guess, "This is a cooking class, and that firefighter is the intruder." You didn't need to see the sign on the door to make a smart guess. The AI was doing the same thing.

2. Thinking Harder Doesn't Always Help (When the Question is Missing)

The researchers tried to make the AIs "think harder" by giving them more time and asking for longer explanations.

The Result: When the AI had the full question, thinking harder made it much smarter.
The Twist: When the AI only had the answers (no question), making it think longer barely improved its score.
Why? It suggests that the AI isn't struggling to "figure it out" with more time; it's using a specific set of skills (like pattern recognition and elimination) that work well immediately. It's not a flaw; it's a different way of solving the puzzle.

3. Not All Shortcuts Are Bad

The paper argues that we shouldn't just throw away these "question-less" tests.

Bad Shortcuts: If the AI picks an answer just because it's the only one with a number in it, that's a flaw. The test is broken.
Good Shortcuts: If the AI picks an answer because it logically eliminated the others based on real-world knowledge (e.g., "Biology doesn't work that way"), that is actually a skill. It shows the AI understands the concepts behind the answers, even without the prompt.

4. The Big Lesson: Fix the Test, Don't Blame the Student

The authors propose a new way to look at AI benchmarks (the tests we use to grade AIs).

Before: If an AI got a question right without seeing the question, we said, "The test is broken" or "The AI is cheating."
Now: We should look at the AI's reasoning.
- If the AI used a shallow trick (like "I picked A because it's the first letter"), then yes, the test is flawed (maybe the answers are too obvious).
- If the AI used deep reasoning (like "I eliminated B and C because they contradict each other"), then the AI is actually smart, and the test is fine.

The Final Metaphor:
Imagine a teacher gives a student a multiple-choice test but accidentally covers the question with a piece of tape.

If the student guesses "A" because it's the first letter, the student is just guessing.
But if the student looks at the answers, realizes three are about "apples" and one is about "cars," and correctly guesses the question was about fruit, that student is actually very smart.

This paper tells us: Don't just look at the score. Look at the reasoning. Sometimes, what looks like cheating is actually a display of impressive, flexible intelligence. And sometimes, it is a flaw in the test itself. The reasoning trace is the tool that helps us tell the difference.

1. Problem Statement

Large Language Models (LLMs) have become proficient at Multiple-Choice Question Answering (MCQA) by generating step-by-step reasoning traces before answering. However, a significant concern exists: LLMs often achieve high accuracy on MCQA tasks without using the question, relying solely on the answer choices (a "choices-only" setting).

The Concern: Previous work suggests this "choices-only" success stems from shallow shortcuts (e.g., exploiting statistical artifacts, keyword matching, or dataset biases) rather than genuine reasoning. This implies that even models with advanced reasoning capabilities might be "cheating" by bypassing the intended cognitive task.
The Gap: It remains unclear whether choices-only success is always a flaw or if reasoning models can employ non-shallow strategies (e.g., inferring missing questions, abductive reasoning) that still utilize skills the MCQ was designed to test. Furthermore, the impact of Test-Time Reasoning (TTR)—scaling compute at inference time—on these partial-input strategies has not been systematically studied.

2. Methodology

The authors conducted a comprehensive empirical study involving 12 state-of-the-art LLMs across three major benchmarks (ARC, MMLU, Super GPQA).

Experimental Setup

Models: 12 LLMs from 6 families (Gemini 2.5, GPT-5/4.1, Claude 3.5/4, Cohere Command, DeepSeek, Qwen3).
Settings:
1. Full Input: Standard MCQA (Question + Choices).
2. Choices-Only: Model sees only the choices ( $C$ ) and must infer the correct answer ( $a$ ) without the question ( $q$ ).
Reasoning Conditions:
- Base: Direct selection without reasoning.
- Reason: Step-by-step reasoning traces generated before the final answer (Chain-of-Thought or API-native reasoning).
Metrics: Accuracy compared against random guessing and across reasoning lengths (Low, Medium, High effort).

Analysis Techniques

Faithfulness Checks: Before analyzing strategies, the authors verified that reasoning traces were "faithful" (i.e., the reasoning actually supported the final answer and wasn't just post-hoc rationalization). They used three checks:
- Consistency of answers with/without TTR.
- Ability of a separate model (GPT-5) to predict the selected answer from the trace (>90% accuracy).
- Detection of intentional perturbations (e.g., duplicate options) in the trace.
Qualitative Coding: A human-in-the-loop qualitative analysis of 180 reasoning traces (correct and incorrect) to categorize reasoning strategies.
Statistical Regression: Logistic regression to determine which strategies predict success or failure in the choices-only setting.

3. Key Contributions

First Analysis of TTR on Choices-Only: The paper is the first to systematically evaluate how scaling test-time reasoning affects accuracy when the question is missing.
Strategy Taxonomy: It identifies a spectrum of strategies used by LLMs in choices-only settings, distinguishing between "shallow shortcuts" and "less problematic" reasoning skills.
Benchmark Refinement Proposal: It proposes a methodology for evaluators to use reasoning trace analysis to distinguish between flawed dataset items and genuine model capabilities, offering a path to improve MCQA benchmark quality.

4. Key Results

A. Impact of Test-Time Reasoning (TTR)

Full Input: TTR significantly boosts accuracy in 25 out of 36 model-dataset combinations.
Choices-Only: TTR provides modest gains (improving accuracy in only 15/36 cases).
Scaling Effect: Increasing the length of reasoning traces (via "Low/Medium/High" effort settings) results in only a slight increase in choices-only accuracy. This suggests that LLMs do not fundamentally change their approach to solving choices-only problems simply by thinking longer; they likely use similar underlying strategies regardless of trace length.

B. Strategy Analysis (The Core Finding)

The qualitative analysis revealed that choices-only success is not monolithic. The authors categorized strategies into five types (Table 1 in the paper):

Strategy	Description	Problematic?
SHALLOW	Simple cues (e.g., "1.5 is the only messy number").	Yes (Bypasses task).
INCONS	Trace contradicts the answer.	Yes (Unfaithful).
FACT	Recalling facts relevant to choices.	No (Uses intended skills).
ELIM	Discarding inaccurate options based on logic.	No (Uses intended skills).
PATTERNS	Naming properties/patterns in choices.	No (Uses intended skills).
INFER Q	Guessing the original question and answering it.	No (Advanced reasoning).

Key Insight: While LLMs do use shallow shortcuts, they frequently employ less problematic strategies.
- Inferring the Question: In successful choices-only attempts, models correctly inferred the original question's topic 83% of the time (ARC) and 77% (MMLU).
- Abductive Reasoning: Models successfully used abductive reasoning to hypothesize the missing question and solve it, a skill that goes beyond what standard MCQs test.
Regression Findings: The presence of SHALLOW and INCONS strategies strongly predicted failure. Conversely, correct answers often relied on FACT, ELIM, and INFER Q.

C. Benchmark Implications

Dataset Flaws: The authors demonstrated that some "choices-only" successes are due to poor MCQ design (e.g., one option being an outlier in content, like a non-kitchen item among kitchen items).
Intervention: By rewriting such questions to ensure homogeneity (removing the outlier), the model's choices-only accuracy dropped, proving that the previous success was a data artifact, not a reasoning failure.

5. Significance and Conclusion

The paper challenges the prevailing view that partial-input success is always a flaw.

For Researchers: Choices-only accuracy alone is an insufficient metric to label a model or dataset as flawed. High choices-only scores can indicate impressive capabilities (like inferring missing context) rather than just data leakage. Reasoning traces are essential to disentangle "cheating" from "strategic reasoning."
For Benchmark Creators: The study provides a concrete workflow for improving MCQA benchmarks:
1. Run models in choices-only mode.
2. Analyze reasoning traces to identify if success stems from shallow shortcuts or genuine inference.
3. If traces reveal shallow shortcuts (e.g., exploiting outliers), rewrite the question to remove the artifact.
Future Directions: The authors advocate for using reasoning traces as "soft evidence" to refine evaluation protocols, ensuring that benchmarks test the intended cognitive skills rather than allowing models to exploit dataset biases.

In summary, the paper argues that while LLMs can and do take shortcuts, test-time reasoning often reveals a more nuanced picture where models are strategically inferring missing information, suggesting that current MCQA benchmarks may need refinement rather than the models needing to be "fixed" to stop thinking.