💻 computer science

Automated Scoring of Handwritten Mathematical Problem Posing Using Large Language Models

This study demonstrates that the large language model Gemini 3.5 Flash can achieve reliable automated scoring of handwritten middle school mathematical problem-posing tasks when using a zero-shot prompting approach with a structured rubric, particularly for whole number problems and when utilizing revised OCR text inputs.

Original authors: Tuğrul Kar, Tarık Kışla, Muhammed Ali Kayici, Burak Aydın

Published 2026-07-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Tuğrul Kar, Tarık Kışla, Muhammed Ali Kayici, Burak Aydın

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher with a mountain of homework to grade. You know that asking students to create their own math problems is a superpower for learning—it builds creativity and deep understanding. But grading these creative, handwritten answers is a nightmare. Every student writes differently, some make up impossible scenarios, and some just write nonsense. It takes forever, and even human teachers can disagree on what a "good" problem looks like. This is where Artificial Intelligence (AI) steps in, promising to be a tireless grading assistant. But here's the catch: AI is great at reading printed text, but can it actually understand messy, handwritten math scribbles and tell the difference between a brilliant student question and a confusing mess? This question sits at the intersection of education and computer science, exploring whether machines can learn to "think" like a math teacher when evaluating student creativity.

The researchers in this study decided to put a very smart AI, called Gemini 3.5 Flash, to the test. They wanted to see if this AI could read handwritten math problems created by 235 middle school students and grade them as accurately as human teachers do. To do this, they set up a "taste test" with two different scenarios: one involving whole numbers (like driving distances) and another involving fractions (like sharing a cake). They gave the AI two versions of the same handwritten text: one that was read directly by the computer's eyes (which sometimes makes mistakes with messy handwriting) and one that had been carefully corrected by human experts. They also tried giving the AI the grading rules in two different orders—starting from the lowest score and going up, or starting from the highest and going down—to see if the order of instructions changed the AI's mind.

The results were a mix of "great news" and "needs work." The study found that when the AI used the expert-corrected text, it agreed with human teachers much better than when it used the raw, messy computer reading. However, the type of math problem mattered a lot. The AI was much more accurate when grading problems about whole numbers (the driving task) compared to problems about fractions (the cake task). It seems the AI struggled more with the nuances of fractions, often giving high scores to answers that were mathematically correct but didn't actually fit the story the student was telling. Interestingly, the order in which the grading rules were presented to the AI didn't matter at all; whether the AI started from the bottom or the top of the rubric, its performance stayed the same.

In short, the study suggests that AI can be a reliable helper for grading handwritten math problems, but it isn't perfect yet. It works best when the handwriting is clear and the math involves whole numbers. The researchers found that simply feeding the AI better text (by having humans fix the computer's reading errors) made a noticeable difference, but the AI still needs help understanding the "story" behind the math, especially when fractions are involved. While the AI didn't get confused by the order of the instructions, it did get tripped up by the complexity of the math content. The authors conclude that while AI holds substantial potential to help teachers, we need to be careful about how we use it, especially for more complex topics like fractions, and we should always double-check that the computer is reading the student's handwriting correctly before it starts grading.

Technical Summary: Automated Scoring of Handwritten Mathematical Problem Posing Using Large Language Models

Problem Statement
While teachers recognize the potential of artificial intelligence (AI) for assessing mathematical problem-posing responses, empirical evidence regarding the reliability of automated scoring (AS) in this specific domain remains limited. Problem posing generates diverse, open-ended responses that are difficult to analyze systematically, often leading to inconsistencies and subjectivity in human scoring. Although Large Language Models (LLMs) have become dominant tools for AS, their application to handwritten mathematical problem posing—particularly involving non-English languages and complex content like fractions—has not been rigorously validated. Existing studies often rely on single-task designs or exemplar-based rubrics that may not translate to authentic classroom settings. Furthermore, it is unclear how task types (e.g., whole numbers vs. fractions) and prompt engineering variables (e.g., rubric ordering) influence the agreement between human raters and LLMs.

Methodology
The study employed a quantitative design to examine the agreement between human raters and the Gemini 3.5 Flash LLM in scoring handwritten student responses.

Participants and Tasks: Data was collected from 235 middle school students (Grades 6 and 7) in Türkiye. Students completed two distinct problem-posing tasks: a "Cake" task involving fractions and a "Drive" task involving whole numbers. Each student generated three problems per task.
Data Processing:
- Human Coding: Two experienced raters scored responses using a consensus-oriented approach based on a refined seven-level rubric (ranging from 0: No response to 6: Sufficient). The rubric incorporated criteria for task relatedness, solvability, and logical consistency.
- OCR and Input Variations: Handwritten responses were digitized using Gemini's Optical Character Recognition (OCR). Two input datasets were created for each response:
  - i1: Original OCR output generated solely by the LLM.
  - i2: Expert-revised OCR output, where domain experts corrected transcription errors (e.g., misread fractions or names).
- Automated Scoring (AS): The Gemini 3.5 Flash model was used via a zero-shot prompting approach. The model was assigned the persona of an expert assessment assistant and instructed to score based strictly on the provided rubric.
- Prompt Variations: The study manipulated the ordering of rubric criteria in the prompts:
  - p1 (Ascending): Evaluation from lowest (0) to highest (6) category.
  - p2 (Descending): Evaluation from highest (6) to lowest (0) category.
- Experimental Conditions: This resulted in four AS scenarios per task (ASi1p1, ASi1p2, ASi2p1, ASi2p2).
Analysis:
- Agreement Metrics: The study computed Quadratic-Weighted Kappa (QWK), Linear-Weighted Kappa (LWK), Pearson correlation, distance correlation, and Percent of Tertium Quid (PTQ) to measure agreement between human and AS scores.
- Multilevel Modeling: A three-level linear model was used to predict the absolute difference between human and AS scores. The dependent variable was the absolute score difference. Predictors included Task Type (Cake vs. Drive), Input Type (i1 vs. i2), and Prompt Direction (p1 vs. p2), along with their interactions. Random effects were nested within answers and students.

Key Results

Overall Agreement: The study found substantial potential for zero-shot prompting with well-structured rubrics. The average absolute score difference between human raters and AS was 0.43 points on a 0–6 scale.
Impact of Input Quality: Agreement was significantly stronger when using revised OCR inputs (i2) compared to original OCR outputs (i1). The input type effect was significant ( $p = .002$ ), though the magnitude of improvement was modest (a reduction of 0.06 points in absolute difference).
Impact of Task Type: Task type was the strongest predictor of agreement. The "Drive" task (whole numbers) yielded significantly higher agreement than the "Cake" task (fractions) ( $p < .001$ $p < .001$ ). The absolute difference was 0.17 points lower for the Drive task.
- Drive Task: Strongest agreement was observed with revised inputs and ascending prompts (QWK = 0.849).
- Cake Task: Weakest agreement occurred with original inputs and ascending prompts (QWK = 0.684).
Impact of Prompt Ordering: The direction of the rubric (ascending vs. descending) had no significant effect on human-machine agreement ( $p = .428$ ). No significant interactions were found between task, input, and prompt direction.
Error Analysis: Discrepancies in the Cake task often stemmed from the LLM assigning high scores to mathematically valid but context-independent responses (e.g., calculating the mean of speeds without integrating the story context) or misinterpreting whether an answer was explicitly available in the data. OCR errors occasionally led to misclassification of task relevance or logical inconsistencies.

Key Contributions

Empirical Evidence in Problem Posing: The study provides one of the first empirical validations of LLM-based AS specifically for mathematical problem posing, extending previous findings from problem solving to the generation of problems.
Zero-Shot Viability: It demonstrates that zero-shot prompting, when paired with a structured rubric, can achieve reliability levels comparable to human raters without the need for supervised fine-tuning or few-shot training.
Task-Dependent Reliability: The research highlights that AS reliability is not uniform across mathematical domains; it performs significantly better with whole numbers than with fractions, suggesting content-specific limitations in current LLM capabilities.
Prompt Engineering Insights: The study challenges the notion that rubric ordering (ascending vs. descending) significantly impacts scoring accuracy in this context, suggesting that logically structured frameworks may mitigate response-order biases in advanced LLMs.

Significance and Claims
The authors claim that their findings offer empirical support for the use of LLMs to assist teachers in evaluating student problem-posing tasks, potentially reducing assessment workload and enabling more frequent integration of problem posing into classrooms. The study suggests that while AS is robust for whole-number tasks, caution is required when applying it to fraction-based tasks or contexts requiring deep contextual integration. The authors emphasize that ensuring high-quality input (via OCR correction) and refining prompt design to address specific coding challenges (such as context integration and data sufficiency) are critical for maximizing the validity of automated scoring in mathematics education. The paper concludes that while AS shows promise, further validation across diverse topics and languages is necessary before broad generalization.

More like this