Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks

This study evaluates eleven general-purpose and education-specific AI tools, finding that they achieve only moderate accuracy (63%) in classifying the cognitive demand of mathematical tasks due to a systematic bias toward middle-level categories and a tendency to prioritize surface textual features over underlying cognitive processes, thereby limiting their immediate reliability for teacher planning without improved prompt engineering or tool development.

The Gist

Imagine you are a head chef (a teacher) running a busy kitchen. You have a massive menu of recipes (math problems) that range from simple "toast a slice of bread" tasks to complex "create a 12-course molecular gastronomy feast" challenges.

Your job is to sort these recipes into the right bins so you can serve the right level of challenge to your diners (students). If you give a beginner a complex feast, they'll burn out. If you give an expert just toast, they'll get bored.

Recently, a new assistant arrived in your kitchen: Artificial Intelligence (AI). You asked, "Can you look at these recipes and tell me which ones are simple and which ones are complex?"

This paper is the report card on how well that AI assistant did its first day on the job.

The Setup: The "Cognitive Demand" Menu

The researchers used a famous framework called the Task Analysis Guide (TAG). Think of this as a four-star rating system for math problems:

1. Memorization (1 Star): Just reciting facts. Like saying "2 + 2 = 4" without thinking.
2. Procedures Without Connections (2 Stars): Following a recipe step-by-step without knowing why the ingredients work. Like mixing cake batter because the box says so.
3. Procedures With Connections (3 Stars): Using the recipe to understand why the cake rises. You are connecting the steps to the science of baking.
4. Doing Mathematics (4 Stars): Creating a brand new recipe from scratch. You have to figure out the ingredients, the method, and the logic yourself. It's messy, hard, and requires deep thinking.

The Experiment: The "Out-of-the-Box" Test

The researchers didn't try to "train" the AI or give it special instructions. They just handed 11 different AI tools (some general ones like ChatGPT, some made specifically for schools like Khanmigo) a stack of 12 math problems and said, "Here is the rulebook. Tell me which star rating each problem gets."

They wanted to see what happens when a teacher uses AI right now, without spending hours learning how to "prompt engineer" (give it perfect instructions).

The Results: The AI is a "Safe-Middle" Assistant

The results were a mix of "not bad" and "pretty concerning."

1. The Average Score: 63%
The AI got about 6 out of 10 problems right. That's better than guessing, but far from the 100% accuracy a teacher needs to trust it blindly.

2. The "Middle-of-the-Road" Bias
This was the biggest finding. The AI was terrified of the extremes.

• When a problem was clearly simple (Memorization), the AI often said, "Oh, it's a 2-star procedure."
• When a problem was clearly hard (Doing Mathematics), the AI often said, "That's just a 3-star procedure."
• The Analogy: Imagine a weather forecaster who is afraid to say "It's a tornado" or "It's a drought." Instead, they just keep saying, "It's a cloudy day." The AI kept pushing everything into the middle categories (Procedures) because it felt safer there.

3. The "Surface-Level" Trap
The AI was great at spotting keywords but terrible at understanding thinking.

• Example: If a problem said "Show your work," the AI thought, "Oh, showing work means it's a simple procedure!"
• Reality: Sometimes "showing your work" means explaining a complex, creative solution. The AI looked at the surface text (the words) rather than the deep thinking (the brain power) required.

4. General vs. Specialized Tools
You might think the AI tools built specifically for schools (like Magic School or Khanmigo) would be better than the general ones (like ChatGPT or Grok).

• The Twist: They weren't. The general-purpose tools performed just as well (or just as poorly) as the school-specific ones. Being "education-branded" didn't make them smarter at this specific task.

The "Why" Behind the Mistakes

The researchers dug into the AI's "reasoning" (its explanation of why it gave a certain rating). They found the AI was overconfident but wrong.

• The "Plausible Lie": The AI would give a very convincing, professional-sounding explanation for why a hard problem was actually easy. It sounded like a teacher, but it was missing the point.
• The "Task G" Failure: There was one specific problem (Task G) where the AI failed 3 out of 4 times. It was a problem that looked like a real-world scenario but had a very specific instruction ("Set up a proportion"). The AI got confused by the real-world story and missed the specific instruction that made it a simple task.

What This Means for Teachers

So, should you fire your human assistant and hire the AI? No.

• Not Ready for Prime Time: You cannot let the AI grade your lesson plans or sort your math problems on its own. It will misclassify the hardest and easiest problems, which are the ones you need to get right most.
• A Good "Second Opinion": The AI is great as a drafting assistant. You can ask it, "Hey, what do you think the difficulty of this problem is?" and then you check its work. It can save you time by doing the first pass, but you must be the final judge.
• The "Prompt" Problem: The study used a very basic prompt. If teachers learn how to talk to the AI better (giving it examples, asking it to think step-by-step), the scores might go up. But right now, out of the box, it's a bit clumsy.

The Bottom Line

AI is like a new intern in the kitchen. It's eager, it knows a lot of facts, and it can read a recipe book. But it doesn't yet have the "taste" or the "intuition" to know if a dish is truly simple or truly complex. It tends to play it safe and call everything "medium."

Until we teach it better how to think (not just read), teachers need to keep their aprons on and do the final tasting themselves.

Technical Summary

Here is a detailed technical summary of the paper "Baseline Performance of AI Tools in Classifying Cognitive Demand of Mathematical Tasks."

1. Problem Statement

Teachers face significant time constraints in adapting high-quality mathematics curricula to meet individual student needs while maintaining cognitive rigor. The cognitive demand of a mathematical task (ranging from rote memorization to complex reasoning) is a critical predictor of student learning outcomes. However, accurately identifying and modifying tasks to ensure appropriate cognitive demand requires substantial pedagogical expertise and time.

While Artificial Intelligence (AI) and Large Language Models (LLMs) are increasingly proposed as tools to reduce teacher workload, there is a lack of systematic evidence regarding their ability to perform core pedagogical tasks. Specifically, it is unknown whether current AI tools can accurately classify the cognitive demand of mathematical tasks using established research frameworks, or if they merely mimic surface-level features without understanding underlying cognitive processes.

2. Methodology

2.1 Theoretical Framework

The study utilizes the Task Analysis Guide (TAG), a widely validated framework developed by Stein and Smith (1998) from the QUASAR project. The TAG categorizes mathematical tasks into four levels of cognitive demand:

1. Memorization: Reproduction of facts/rules without conceptual connection.
2. Procedures without Connections: Algorithmic execution to produce correct answers without conceptual understanding.
3. Procedures with Connections: Use of procedures to develop deeper conceptual understanding.
4. Doing Mathematics: Complex, non-algorithmic thinking requiring self-monitoring and multiple solution paths.

2.2 Data and Task Selection

• Tasks: 12 mathematics tasks were selected, representing all four TAG categories (3 tasks per category). These tasks were previously analyzed and labeled by human experts trained in the TAG.
• AI Tools: 11 tools were tested:
  • 6 General-Purpose: ChatGPT, Claude, DeepSeek, Gemini, Grok, Perplexity.
  • 5 Education-Specific: Brisk, Coteach AI, Khanmigo, Magic School, School.AI.

2.3 Experimental Procedure

• Prompting Strategy: The study employed an "out-of-the-box" approach to establish a realistic baseline for teachers. A single standardized prompt was used across all tools and tasks, uploading the TAG framework and a specific task file. No prompt engineering, few-shot learning, or chain-of-thought optimization was applied.
• Prompt: "Based on the 'TAG.docx' I uploaded, determine whether the math task 'Task[X].docx' uploaded is high-cognitive demand or low-cognitive demand. Give detailed reasoning and output the detailed analysis with dimensions in a table."
• Evaluation:
  • Accuracy: Comparison of AI classifications against expert labels.
  • Error Analysis: Human experts deconstructed AI reasoning line-by-line, coding whether the AI addressed specific TAG characteristics correctly (Score 0–3).
  • Bias Analysis: Examined distribution patterns across the four categories.

3. Key Results

3.1 Overall Accuracy

• Average Accuracy: AI tools achieved an average classification accuracy of 62% (range: 9% to 100%).
• Comparison: This is significantly above chance (25%) but far below the reliability required for autonomous educational planning.
• Tool Performance:
  • Top Performer: DeepSeek (General-Purpose) at 83%.
  • Education-Specific vs. General: Education-specific tools (avg 63%) did not significantly outperform general-purpose tools (avg 61%).
  • Lowest Performers: ChatGPT and Claude (General) and Khanmigo (Education) scored around 50%.

3.2 Systematic Biases and Patterns

• Central Tendency Bias: AI tools exhibited a strong bias toward the middle categories (Procedures with/without Connections).
  • 77% of all classifications fell into the two middle categories.
  • Only 11% were classified as Memorization and 11% as Doing Mathematics.
  • Tools struggled significantly with the extremes (Memorization and Doing Mathematics).
• Task-Specific Variance:
  • Procedures without Connections: 100% accuracy (Tasks D, H, J). These tasks had clear, explicit algorithmic instructions.
  • Doing Mathematics: Extremely low accuracy (e.g., Task K was only 9% accurate). Tools failed to recognize non-algorithmic complexity and real-world modeling requirements.
  • Memorization: Low accuracy (avg 44%), often misclassified as "Procedures without Connections" because the AI detected the use of a procedure rather than the lack of conceptual depth.

3.3 Error Analysis and Reasoning Quality

• Surface Feature Over-weighting: AI tools consistently prioritized surface textual features (e.g., the presence of the word "algorithm" or "procedure") over the analysis of underlying cognitive processes.
• Flawed Reasoning: Even when tools arrived at the correct broad category (High vs. Low demand), their reasoning was often flawed.
  • Experts found that AI reasoning scores for TAG characteristics rarely reached "fully correct" (Score 3). Most scores were between 1.0 and 2.0 (partially correct or addressed but incorrect).
  • Example: In misclassifying a "Doing Mathematics" task, tools often focused on the transparency of the cost structure in a word problem rather than the requirement for students to construct their own equations and justify decisions.
• Broad-Level Errors: Only 5% of classifications were "broad-level" errors (misidentifying High vs. Low demand), but these represented fundamental misunderstandings of the task nature.

4. Key Contributions

1. First Systematic Baseline: Provides the first empirical evaluation of both general-purpose and education-specific AI tools on a core pedagogical task (cognitive demand classification) using a rigorous, research-based framework.
2. Differentiation of Tool Types: Demonstrates that "education-specific" AI tools are not inherently superior to general-purpose LLMs for this specific task, challenging the assumption that domain-specific fine-tuning guarantees better pedagogical reasoning.
3. Identification of Systematic Biases: Reveals a "central tendency bias" where AI defaults to middle categories when uncertain, and a "surface feature bias" where AI confuses the presence of a procedure with low cognitive demand.
4. Reasoning vs. Accuracy: Highlights that high classification accuracy does not equate to high-quality reasoning. AI tools often provide plausible-sounding but pedagogically incorrect justifications, which could be misleading to novice teachers.

5. Significance and Implications

• Readiness for Deployment: Current AI tools are not ready for autonomous use in assessing cognitive demand. An accuracy rate of ~62% with significant bias toward middle categories poses a risk of teachers inadvertently lowering the rigor of their curriculum or misidentifying task complexity.
• Decision Support Role: AI shows promise as a decision-support system rather than an autonomous classifier. It can flag tasks for human review or provide preliminary drafts that must be verified by expert teachers.
• Need for Prompt Engineering: The "out-of-the-box" performance suggests that optimized prompts (e.g., few-shot learning, chain-of-thought, specific rubric constraints) could significantly improve accuracy. However, this places an additional burden on teachers to learn prompt engineering.
• Future Development: Tool developers must address the tendency to over-weight surface features and under-weight cognitive process analysis. Future iterations need to be trained to distinguish between procedural execution and conceptual engagement more effectively.
• Teacher Expertise: The study reinforces the irreplaceable value of human teacher expertise in curriculum adaptation. AI cannot yet replicate the nuanced judgment required to evaluate the cognitive depth of mathematical tasks.