Technical Summary: Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

Problem Statement

The field of Large Language Model (LLM) evaluation has increasingly assumed that sophisticated prompting techniques—ranging from Chain-of-Thought (CoT) to self-consistency and analogical reasoning—are necessary to unlock superior performance, particularly in Multiple-Choice Question Answering (MCQA). However, existing literature often lacks comprehensive, controlled evaluations comparing these techniques across diverse models and datasets. Many studies claim performance gains from complex prompting without rigorous statistical validation or without accounting for the specific model architectures (e.g., reasoning-enabled vs. non-reasoning models) being tested. This creates a gap in understanding whether prompt complexity genuinely improves reasoning or if simpler baselines are sufficient, and whether current benchmarks are saturated or still offer room for genuine model improvement.

Methodology

The authors introduce ReasonLab, a systematic evaluation framework designed to compare prompting techniques head-to-head under a unified protocol. The study encompasses:

• Scale: Approximately 430,000 evaluations across 27 model configurations, 10 MCQA datasets, and 8 distinct prompting techniques.
• Datasets: Ten diverse MCQA benchmarks covering physics, medicine, mathematics, IoT/time-series, and multi-domain graduate knowledge (e.g., MMLU-Pro, SuperGPQA, AIME 2025). These datasets vary significantly in question length, number of options, and answer-key distribution skew.
• Models: 27 configurations spanning 13 distinct model families (including GPT-5, Claude Sonnet 4, Qwen3, Llama, Mistral, and Granite). The study distinguishes between models with vendor-exposed "reasoning toggles" (configurable thinking budgets) and those without.
• Prompting Techniques: Eight techniques were evaluated:
  1. Baseline: Direct prompt with answer letter only.
  2. CoT-Standard: "Let me think step by step."
  3. CoT-Expert: Role-framed CoT (e.g., "as a reliability engineer").
  4. CoT-Inductive: Inductive reasoning framing.
  5. Self-Plan-and-Solve: Explicit plan generation.
  6. Prompt-Repeat: Repeated direct prompts.
  7. Self-Generate: Generating contextual documents.
  8. Self-Analogical: Generating related MCQA exemplars.
• Evaluation Strategy:
  • Matched Comparison: A strict 100-cell comparison (10 datasets × 10 non-reasoning model configurations) where every technique is tested on the exact same questions and models to enable paired statistical testing (t-tests).
  • Raw Comparison: Aggregated accuracy across all available cells, acknowledging potential imbalances in model coverage.
  • Metrics: Accuracy, token costs (reasoning vs. output), and Elo ratings derived from pairwise model comparisons on individual questions.

Key Contributions and Findings

1. The Simplicity Paradox: Baseline vs. Complex Prompting

On the matched set of 10 non-reasoning model configurations, the baseline prompt (direct answer) consistently outperformed or matched complex reasoning scaffolds.

• Performance: The baseline achieved a mean accuracy of 52.08%. It was outperformed only by CoT-Expert (55.71%) and CoT-Inductive (55.40%), both of which are minimal role-framing variants rather than multi-step reasoning frameworks. The gain was small (~3 percentage points) but statistically significant.
• Underperformance of Complex Scaffolds: Elaborate techniques like Self-Plan-and-Solve, Self-Generate, and standard CoT underperformed the baseline by significant margins (up to 9.45 pp). Self-Analogical performed worst (21.38%), largely due to format-collapse failures on smaller models where the model failed to parse the complex output structure.
• Conclusion: For models without native reasoning capabilities, prompt complexity does not yield better performance; simple direct prompting is often superior or equivalent.

2. The Qwen Surprise: Parameter Count vs. Performance

The study challenges the assumption that parameter count is the primary predictor of performance at the frontier.

• Elo Leaderboard: Qwen3-30B-A3B-Thinking-2507 (approx. 30B total parameters, ~3B active) achieved the highest Elo rating (1657), narrowly beating larger models like GPT-OSS-120B (4× larger) and Llama-3.1-405B (13× larger).
• Implication: While the top 8 models clustered within a tight 14.3 Elo points, the fact that a 30B model outperformed 120B+ and 405B+ models suggests that parameter count is a poor predictor of MCQA capability in the current generation of models.

3. Reasoning Budgets: The "Enable" vs. "Scale" Trade-off

The study analyzed the impact of explicit reasoning token budgets (nothink, low=1k tokens, medium=8k tokens) on models with reasoning toggles.

• Enabling Reasoning: The most significant performance gains came from simply enabling reasoning (switching from nothink to medium). For GPT-5 variants, this yielded gains of +19.7 to +22.5 pp. For Claude Sonnet 4, the gain was smaller (+3.6 pp), and interestingly, the low setting (1k tokens) underperformed nothink, suggesting small budgets may disrupt implicit reasoning.
• Scaling Budgets: Increasing the budget within the "on" state (from low to medium, an 8× increase) yielded diminishing returns, adding only +1.0 to +4.1 pp at the cost of hundreds of additional reasoning tokens per question.
• Conclusion: The value of test-time compute is highly model-dependent; the primary lever is enabling the reasoning mechanism, not arbitrarily scaling the token budget.

4. Dataset Difficulty and Saturation

Contrary to claims of benchmark saturation, the study found substantial headroom for model improvement.

• Difficulty Spread: There is a 47.5 pp spread in mean accuracy between the easiest dataset (OpenBookQA, 83.4%) and the hardest (SuperGPQA, 35.8%).
• Unsaturated Benchmarks: 6 out of 10 datasets had mean accuracy below 70% across all models. Even the best-performing model on each dataset failed to reach 70% on 4 of the 10 datasets (including SuperGPQA, AIME 2025, and FailureSensorIQ).
• Conclusion: The field has not reached saturation; the remaining performance gaps lie in genuine reasoning, specialized knowledge, and temporal understanding rather than prompt engineering.

Significance and Claims

The paper argues that the LLM evaluation community may be overcomplicating prompt engineering. The "Simplicity Paradox" suggests that for non-reasoning models, elaborate prompting techniques often fail to improve performance and can even degrade it due to format issues.

The authors claim that:

1. Prompt optimization has diminishing returns: Substantial performance gaps remain across diverse benchmarks, offering opportunities for genuine model improvements (architecture, training data) rather than further prompt scaffolding.
2. Evaluation needs nuance: The distinction between "enabling reasoning" and "scaling reasoning" is critical, and parameter counts are not reliable proxies for capability.
3. Benchmark limitations: Current MCQA leaderboards are not saturated; the difficulty of existing benchmarks (like SuperGPQA) indicates that models still struggle with complex, real-world reasoning tasks.

The study concludes by calling for a redirection of effort from prompt scaffolding toward base-model capability and harder evaluations, noting that the "victory" of simple baselines over complex methods highlights a need to re-evaluate the necessity of complex prompting strategies in the current era of LLMs.