From Test-taking to Cognitive Scaffolding: A Pedagogical Diagnostic Benchmark for LLMs on English Standardized Tests

This paper introduces ESTBook, a multimodal benchmark of over 10,000 English standardized test questions enriched with cognitive scaffolding and distractor rationales, to shift LLM evaluation from simple accuracy to diagnosing human misconceptions and improving pedagogical reasoning.

The Gist

Imagine you are hiring a tutor to help a student prepare for a big, important exam like the SAT, GRE, or TOEFL.

The Old Way: The "Black Box" Tutor
Until now, most people have tested AI tutors the same way they test a calculator: they ask a question, and if the AI gets the right answer, they give it a gold star. If it gets it wrong, they give it a red X.

The problem with this approach is that it's like judging a chef only by whether the final dish tastes good, without ever watching how they chopped the vegetables or seasoned the soup. An AI might get the right answer by pure luck, or by guessing, or by using a "shortcut" that works for this one question but would fail miserably on the next one. It might arrive at the correct answer while completely misunderstanding the math or the logic along the way.

The New Way: The "Cognitive X-Ray"
This paper introduces a new way to test AI, called ESTBOOK. Instead of just looking at the final answer, the researchers built a system that acts like an X-ray machine for the AI's brain. They break every test question down into a specific "cognitive trajectory"—a step-by-step map of how a human expert actually solves the problem.

Think of it like a GPS for problem-solving. Instead of just saying "You arrived at the destination," the GPS now says:

1. Step 1: Did you correctly read the map? (Understanding the question)
2. Step 2: Did you choose the right route? (Formulating the math or logic)
3. Step 3: Did you drive the car correctly? (Doing the actual calculation)
4. Step 4: Did you avoid the potholes? (Ignoring the tricky wrong answers)

What They Found
The researchers tested the world's smartest AI models (like GPT-5, Claude, and Gemini) on over 10,000 real exam questions covering text, math, charts, and audio. Here is what they discovered:

• The "Smart but Flaky" Problem: The AIs are great at the beginning and the end. They can usually understand the question and write a good final sentence. But they often crash in the middle. They might set up the math equation perfectly but then make a silly arithmetic mistake, or they might get distracted by a "trick" answer that sounds right but is actually wrong.
• The Distractor Trap: In a multiple-choice test, the wrong answers (distractors) are designed to catch common human mistakes. The study found that AIs are surprisingly bad at spotting these traps. If a wrong answer sounds "plausible," the AI often accepts it, even if the logic is broken. It's like a student who sees a word they recognize in a wrong answer and thinks, "That sounds right!" without checking the context.
• Multimodal Confusion: When the test involves mixing different types of information—like reading a paragraph while looking at a complex graph—the AIs get confused. They often mix up the text with the numbers, like trying to read a recipe while looking at a picture of a cake and getting the ingredients wrong.

The Fix: Teaching the AI to "Show Its Work"
The paper doesn't just point out the flaws; it offers a way to fix them. The researchers found that if they force the AI to follow a strict, step-by-step checklist (a "cognitive scaffold") before giving an answer, the performance jumps significantly.

• Analogy: Imagine a student who rushes to write an essay. They get the main idea but mess up the grammar. If you force them to first write an outline, then check their grammar, and then write the essay, the final result is much better.
• The Result: By using these specific "mitigation strategies" (like forcing the AI to quote the text before answering, or to write out the math equation before calculating), the AI became much more reliable and less likely to fall for the trick questions.

The Bottom Line
This paper argues that for AI to be a truly useful tutor, we can't just care about the final score. We need to see the steps. Just as a human teacher needs to know where a student is struggling (is it the vocabulary? the math? the logic?) to help them improve, we need to diagnose AI at the specific step where it fails.

The researchers built a massive new toolkit (ESTBOOK) that does exactly this, turning the AI from a "black box" that just guesses answers into a transparent system where we can see exactly how it thinks, where it gets stuck, and how to teach it to think more like a human expert.

Technical Summary

1. Problem Statement

Current evaluations of Large Language Models (LLMs) in educational contexts, particularly on English Standardized Tests (ESTs) like the SAT, GRE, GMAT, TOEFL, and IELTS, rely predominantly on binary outcome accuracy (i.e., whether the final answer is correct). This approach is insufficient for deploying LLMs as intelligent educational tutors because:

• Lack of Pedagogical Utility: A model can arrive at the correct answer through flawed intermediate logic or hallucinations, rendering it useless for explaining concepts to students.
• Inability to Diagnose Misconceptions: Effective tutoring requires identifying why a distractor option is incorrect and diagnosing specific human cognitive traps (e.g., partial truth, execution errors).
• Black-Box Reasoning: Traditional benchmarks treat problem-solving as a monolithic task, failing to isolate specific reasoning bottlenecks (e.g., visual parsing vs. arithmetic execution).

The paper argues that to transition LLMs from "test-takers" to "tutors," evaluation must shift from final output accuracy to step-by-step cognitive trajectory analysis.

2. Methodology: ESTBOOK and the Cognitive Diagnostic Framework

The authors introduce ESTBOOK, a multimodal pedagogical diagnostic benchmark, and a formalized Cognitive Trajectory Framework.

A. The Dataset: ESTBOOK

• Scale & Scope: Contains 10,576 questions across 29 distinct task types from five major exams (SAT, GRE, GMAT, TOEFL, IELTS).
• Multimodality: Includes text, mathematical symbols, images, tables, and audio (transcribed via Whisper).
• Annotation Strategy: Unlike standard datasets, ESTBOOK is enriched with:
  • Formalized Cognitive Trajectories: Each question is mapped to a specific sequence of cognitive sub-skills (nodes) required to solve it.
  • Distractor Rationales: Incorrect options are annotated with the specific "cognitive trap" they represent (e.g., "Partial Truth," "Execution Error," "Out of Scope").
  • Non-Generative Pipeline: Annotations were created using deterministic NLP techniques (dependency parsing, rule-based mapping) and human-in-the-loop validation to avoid data contamination from generative LLMs.

B. The Cognitive Trajectory Framework

The authors model problem-solving as a traversal through a structured graph of cognitive nodes ($C = \{c_1, c_2, \dots, c_n\}$). They categorize tasks into three pedagogical domains:

1. Knowledge-Intensive Retrieval (Lexical & Structural):
   • Sub-skills: Syntactic parsing, rule matching, semantic resolution.
   • Example: GRE Text Completion, SAT Writing.
2. Reasoning-Intensive Execution (Multimodal & Quantitative):
   • Sub-skills: Analytical goal setting, visual parsing, mathematical formulation, symbolic computation.
   • Example: GRE Data Interpretation, SAT Math.
3. Hybrid Integration (Semantic Extraction & Inference):
   • Sub-skills: Intent identification, evidence extraction, constraint application, comparative evaluation.
   • Example: TOEFL Reading, GMAT Critical Reasoning.

C. Evaluation Metrics

Instead of simple accuracy, the framework uses node-level metrics tailored to the cognitive step:

• Extraction/Localization: Intersection over Union (IoU) and Token-level F1.
• Mathematical/Formulation: Symbolic Equivalency (using Computer Algebra Systems like SymPy) to handle algebraic variations.
• Execution: Normalized RMSE for numeric outputs.
• Generative/Deductive: BERTScore for semantic fidelity.

3. Key Contributions

1. ESTBOOK Benchmark: The first large-scale, multimodal dataset for ESTs that moves beyond answer keys to include structured reasoning trajectories and distractor rationales.
2. Cognitive Diagnostic Framework: A novel methodology that decomposes LLM reasoning into granular cognitive nodes, allowing for the precise isolation of failure points (e.g., distinguishing between a model that understands the problem but fails arithmetic vs. one that fails to parse the visual input).
3. Targeted Mitigation Strategies: The paper proposes and validates specific "elicitation" strategies (e.g., Evidence-Anchored CoT, Syntax-First prompts, Table-Alignment constraints) that address specific bottlenecks identified in the framework.

4. Experimental Results

The authors evaluated state-of-the-art Multimodal LLMs (GPT-5, GPT-4V, Claude-Sonnet-4, Llama-4-Scout, Qwen-VL-Max, Gemini-2.5) against human testers.

A. Performance Gaps & Bottlenecks

• Formulation vs. Execution: LLMs generally excel at the initial steps (problem modeling, task identification) with up to 97% accuracy but show significant performance drops in subsequent reasoning and execution steps.
• The "Integration Bottleneck": A critical failure point occurs at Step 2 (binding parsed constraints to representations). Models often hallucinate a valid integration when faced with distractors containing "Partial Truths" or "Faulty Premises."
• Modality Issues: Performance degrades significantly in multimodal tasks (e.g., GMAT Integrated Reasoning) where models fail to align textual clues with tabular data or visual charts.
• Distractor Susceptibility: High accuracy on ground-truth answers does not correlate with robustness against distractors. Models often accept incorrect options that are semantically plausible but logically flawed.

B. Impact of Prompting Strategies

• Chain-of-Thought (CoT): Effective for verbal tasks but can amplify errors if the initial trajectory is flawed (error propagation).
• Tree-of-Thought (ToT): Helpful for search-like tasks but introduces variance and "path explosion" in constrained logical tasks.
• In-Context Learning (ICL): Highly dependent on schema alignment; mismatched examples can bias models.

C. Mitigation Success

Applying bottleneck-specific mitigation strategies significantly improved performance:

• Evidence-Anchored CoT: Improved GRE Reading Comprehension accuracy from 77.8% to 93.5% (GPT-4V).
• Table-Alignment Constraints: Improved GMAT Integrated Reasoning from 13.8% to 59.7% (GPT-4V).
• Symbolic Verification: Boosted GMAT Problem Solving accuracy by over 20 percentage points by forcing explicit equation verification before calculation.

5. Significance and Implications

• Pedagogical Shift: The paper establishes that for AI to be a viable educational tutor, it must demonstrate faithful reasoning and the ability to diagnose misconceptions, not just provide correct answers.
• Diagnostic Precision: The framework allows educators and developers to pinpoint exactly where an LLM fails (e.g., is it a visual parsing error or a math execution error?), enabling targeted model improvement.
• Actionable Interventions: The study proves that simple prompting adjustments (scaffolding) based on cognitive diagnosis can close the performance gap between LLMs and humans in specific domains, making LLMs more reliable for real-world educational deployment.
• Future Directions: The work suggests a move away from monolithic benchmarking toward stepwise evaluation and the development of hybrid systems where LLMs handle planning/reasoning while specialized modules (symbolic solvers, visual parsers) handle execution.

In conclusion, ESTBOOK provides a rigorous diagnostic lens that reveals current LLMs are strong "planners" but weak "executors" and "discriminators" in complex educational scenarios, offering a clear roadmap for building more robust, pedagogically sound AI tutors.