💬 NLP

Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?

This study reveals that the format of questions, including the number of options and specific wording, significantly impacts Large Language Model performance on reasoning tasks, often causing a disconnect between the accuracy of reasoning steps and the correctness of the final answer.

Original authors: Seok Hwan Song, Mohna Chakraborty, Qi Li, Wallapak Tavanapong

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seok Hwan Song, Mohna Chakraborty, Qi Li, Wallapak Tavanapong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to test how smart a new student (an AI) is at solving logic puzzles. You want to see if the student truly understands the math or logic, or if they are just guessing the right answer.

This paper is like a report card from a study where the researchers asked the same logic problems to five different AI students, but they asked the questions in three very different ways:

The "Essay" Style (Short Answer): "Here is a problem. Solve it and tell me the answer."
The "Multiple Choice" Style: "Here is a problem. Pick the right answer from these 5 or 11 options."
The "True or False" Style: "Here is a problem and a specific answer. Is this answer True or False?"

The researchers wanted to know: Does the way you ask the question change how well the AI performs?

Here are the main takeaways, explained simply:

1. The "Guessing Game" Effect

The biggest surprise was that getting the right final answer doesn't always mean the AI did the right work.

The Analogy: Imagine a student taking a multiple-choice test. They might not know how to do the math, but they guess "C" and get it right. If you only look at the bubble sheet, they got a 100%. But if you look at their scratch paper, they wrote nonsense.
The Finding: The AI often "guessed" the right letter in a multiple-choice question even when its reasoning steps were wrong. Conversely, sometimes the AI did the perfect math but picked the wrong letter at the very end because it got confused by the options.
The Lesson: If you only check the final answer, you might think the AI is smarter than it actually is.

2. The "Menu" Problem (Multiple Choice)

When the researchers gave the AI a list of options to choose from, the number of options and the words used mattered a lot.

Too Many Choices: When the list grew from 5 options to 11 options, the AI's performance often dropped. It's like being offered 5 flavors of ice cream vs. 50; the AI got overwhelmed and started guessing more.
The "Something Else" Option: The researchers added a weird option called "Something else" (or "None of the above").
- If "Something else" was the correct answer, the AI often struggled to pick it, even if it did the math right. It seemed to prefer picking a specific number, even if that number was wrong.
- The AI also seemed to have a "favorite seat" bias. It would pick the first option or the last option more often, regardless of whether it was right.

3. The "Yes/No" Trap (True or False)

When asking "True or False" questions, the specific words used changed the results.

True vs. False: The AI was generally better at saying "True" when the answer was correct than saying "False" when the answer was wrong. It seemed to have a bias toward agreeing.
The Wording: Changing the prompt from "True or False" to "Yes or No" made the AI significantly worse at some tasks. It's like how a person might answer "Yes" to "Do you want to go?" but hesitate on "Is it true that you want to go?" The AI is sensitive to these tiny linguistic shifts.

4. The "Instruction" Confusion

The researchers tried to help the AI by adding instructions like, "Solve the problem first, then decide if it's True or False."

The Result: Sometimes this helped, but often it confused the AI or made it perform worse. The AI sometimes got so focused on the instruction that it forgot the actual logic of the problem.

Summary: What Should We Learn?

The paper concludes that we cannot just look at the final score to judge an AI's intelligence.

If you ask an AI a multiple-choice question, it might be "cheating" by guessing the right letter without understanding the logic.
If you ask it a True/False question, the specific words you use can trick it into being less accurate.
To truly know if an AI is smart, you have to check its "scratch paper" (its reasoning steps), not just the final bubble it filled in.

The researchers built a new set of tests (a benchmark) to help future developers understand these quirks, ensuring that when we test AI, we aren't just testing how good it is at guessing, but how good it is at thinking.

1. Problem Statement

While Large Language Models (LLMs) are extensively evaluated on reasoning tasks using diverse benchmarks, there is a lack of understanding regarding how the format of the question (Short Answer, Multiple Choice, True/False) influences model performance. Existing benchmarks often treat these formats as interchangeable, yet factors such as the number of options, the position of the correct answer, and specific word choices may introduce biases or alter the reasoning process.

The authors aim to answer five specific Research Questions (RQs):

Do question types impact Final Selection (FS) accuracy?
Do question types impact Reasoning Accuracy (the correctness of the intermediate steps)?
What are the patterns of incorrect outputs across different question types?
What specific factors in Multiple Choice Questions (MCQs) influence performance?
What specific factors in True/False Questions (TFQs) influence performance?

2. Methodology

Datasets and Tasks

The study evaluates LLMs on two distinct reasoning tasks:

Quantitative Reasoning: Using the GSM8K dataset (grade-school math word problems).
Deductive Reasoning: Using a modified subset of the bAbI dataset (factoid QA with supporting facts).
Data Volume: 300 problems were randomly selected from each dataset. For bAbI, two irrelevant facts were added to each problem to increase complexity.

Question Type Transformation

Each problem was converted into three formats:

Short Answer Questions (SAQ): The model generates the reasoning steps and the final answer.
Multiple Choice Questions (MCQ): The model selects from a list of options.
- Option Counts: Tested with 5 options and 11 options.
- Special Option: Included a "Something else" (SEO) option to test if models can select an answer outside the provided numeric values.
- Correct Answer Positioning: Tested three distributions: Uniform (U), Uniform excluding SEO (U-SEO), and Only SEO (oSEO).
- Distractors: Wrong answers were generated with specific error margins (5% or 20% deviation) or random equation generation.
True/False Questions (TFQ): The model judges a statement.
- Formats: "Question format" ("Is the answer X?") vs. "Statement format" ("The answer is X.").
- Correctness: Tested with both "True" and "False" as the ground truth.
- Instructions: Tested with and without an instruction to "solve first" before answering.
- Wording: Tested "True/False" vs. "Yes/No".

Models Evaluated

Five LLMs were tested:

Closed-source: GPT-4o, GPT-3.5-turbo.
Open-source: Llama-3 8B, Llama-3.2 1B, Gemma 7B.
Configuration: Greedy decoding (temperature = 0), zero-shot setting.

Evaluation Metrics

Final Selection (FS) Accuracy: Automated string matching of the final output against the ground truth.
Reasoning Accuracy: Manual evaluation of the intermediate reasoning steps. A strict criterion was applied: all reasoning steps must be correct for the reasoning to be marked correct.

Statistical Analysis

The two-sided paired Wilcoxon signed-rank test was used to determine statistical significance ( $p < 0.05$ ) between question types.

3. Key Contributions

Comprehensive Benchmark Design: A novel experimental design that systematically varies MCQ option counts (5 vs. 11), correct answer positions, and TFQ phrasing, addressing gaps in previous literature.
Decoupling Metrics: The study explicitly separates Reasoning Accuracy from Final Selection Accuracy, revealing that they do not always correlate.
Empirical Findings: Identification of specific biases in LLMs, such as sensitivity to the word "Something else," the position of correct answers, and the phrasing of TFQs.
Open Source: The authors released their code and benchmark publicly.

4. Key Results

RQ1 & RQ2: Impact of Question Type on Accuracy

FS vs. Reasoning Correlation: There is no consistent correlation between reasoning accuracy and final selection accuracy.
- Observation: Models often guess the correct final answer despite flawed reasoning (especially in MCQs and TFQs). Conversely, models sometimes derive correct reasoning but fail to select the corresponding option (e.g., due to formatting errors or misunderstanding "Something else").
Performance Variance: Significant differences exist in FS accuracy across question types.
- SAQ vs. MCQ: SAQ generally outperforms MCQ, particularly when "Something else" is the correct answer.
- SAQ vs. TFQ: TFQs with "False" as the correct answer often yield higher FS accuracy than SAQs for some models, suggesting models can guess "False" more easily than deriving the exact number.

RQ3: Patterns of Incorrect Outputs

The study identified distinct failure patterns:

Good Guessing: Models output incorrect reasoning steps but guess the correct final answer (common in TFQs).
Proximity: Models calculate a value close to the correct answer and select the nearest option, even if the calculation was slightly off.
SEO Misunderstanding: Models struggle to select "Something else" even when their reasoning indicates the answer is not in the list; they often force a selection of the closest numeric option.
Missing Selection: Models complete reasoning correctly but fail to output the final selection token.

RQ4: Factors Influencing MCQ Performance

Number of Options: Increasing options from 5 to 11 generally decreases FS accuracy, particularly for smaller models (e.g., Gemma, Llama-1B). This suggests models may be guessing rather than calculating.
Correct Answer Position: Models perform best when the correct answer is not "Something else" (U-SEO). Performance drops significantly when "Something else" is the only correct option (oSEO).
Word Choice: Replacing "Something else" with "None of the above" significantly improved FS accuracy for GPT-3.5, indicating sensitivity to option wording.

RQ5: Factors Influencing TFQ Performance

Question vs. Statement: The Statement format ("The answer is X") generally yields higher FS accuracy than the Question format ("Is the answer X?").
True vs. False: Models generally perform better when "True" is the correct answer, though some models (GPT-3.5, Llama-8B) showed better FS accuracy with "False" in quantitative tasks.
Wording (Yes/No vs. True/False): Replacing "True/False" with "Yes/No" caused a statistically significant drop in performance (up to ~26%) for GPT-3.5, unless a "solve first" instruction was provided.

5. Significance and Implications

Benchmarking Validity: The study challenges the reliability of current benchmarks. A model's high score on MCQs may reflect guessing or pattern matching rather than genuine reasoning capabilities.
Evaluation Strategy: Researchers should not rely solely on Final Selection accuracy. Evaluating Reasoning Accuracy is crucial to understand if a model is actually solving the problem or just selecting the right option.
Prompt Engineering: The specific wording of options ("None of the above" vs. "Something else") and the format of TFQs significantly impact performance. Standardized, carefully designed prompts are necessary for fair evaluation.
Future Directions: Future benchmarks should account for option bias, the number of choices, and the specific phrasing of instructions to ensure they measure reasoning rather than selection bias.

In conclusion, the paper demonstrates that how a question is asked fundamentally alters LLM performance, often decoupling the ability to reason from the ability to select the correct answer. This necessitates a more nuanced approach to LLM evaluation and benchmark design.