💻 computer science

AI-supported formative assessment of conceptual understanding in mathematics: LLM-based pre-coding of open student responses

This study demonstrates that prompt-based large language models can reliably identify conceptual categories in open-ended student mathematics responses without fine-tuning, achieving high accuracy for positive features and deep conceptual understanding while highlighting the need for expert-in-the-loop support and linguistically explicit prompts to address challenges in verifying absent features or resolving ambiguity.

Original authors: Corinna Hankeln

Published 2026-09-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Corinna Hankeln

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the classroom, the most valuable moments often happen not when a student gets the right answer, but when they explain how they got there. Teachers have long known that to truly understand a student's mind, they must look past the final number and see the reasoning behind it. This is the heart of formative assessment: a process of checking in on learning while it is happening, so instruction can be adjusted to help the student move forward. However, there is a practical problem. When students write out their thoughts, draw pictures, or create their own word problems, the teacher faces a mountain of text to read and interpret. Doing this deeply for every student takes time that rarely exists in a busy school day. As a result, many teachers end up grading only the final answers, missing the rich insights hidden in the students' explanations.

For years, researchers have tried to build computers that could read these explanations and spot the thinking patterns inside. Early attempts often failed because they relied on rigid rules or required massive amounts of pre-labeled data to learn. The arrival of large language models—powerful computer systems trained on vast amounts of text to understand and generate human language—has reopened the question. These models can follow instructions written in plain English without needing to be retrained for every specific task. But a critical doubt remains: can a machine that is excellent at language actually understand the specific, often subtle, logic of mathematics? Can it tell the difference between a student who truly grasps a concept and one who is just guessing or using the right words for the wrong reasons?

A researcher at TU Dortmund University set out to answer this question in the context of mathematics education. They wanted to know if these new AI tools could reliably identify specific types of student thinking in open-ended math responses, and if they could do so well enough to be useful in a real classroom. The study focused on two main areas: natural numbers and fractions. The researcher gathered thousands of written responses from students in grades five and six. These students were asked to do things like explain why a fraction works a certain way or to create their own word problem for a given math equation. The goal was to see if an AI, guided by a set of instructions written by a human expert, could sort these responses into meaningful categories, such as "shows deep understanding," "has a specific misconception," or "provides no evidence of understanding."

The researcher tested four different open-source AI models, which are versions of the technology that schools can host on their own servers to protect student privacy. They compared the AI's grading against that of human experts. The results were surprisingly positive. Across more than 13,000 coded responses, the AI models achieved a high level of agreement with the human experts, correctly identifying the main categories of student thinking about 82 percent of the time. This performance was consistent across different types of math problems and different models. The AI was particularly good at spotting when a student correctly explained a concept, such as understanding that multiplication involves grouping items together, or when a student made a clear, specific error in their reasoning.

However, the study also revealed where the machines hit a wall. When the task required the AI to perform a simple arithmetic check—like verifying if the numbers in a student's story actually added up correctly—the models struggled. They also had trouble when a student's response was very short or vague, or when the student used language that was ambiguous. In these tricky situations, the AI tended to guess, often assuming the student understood more than they actually did. The researcher found that the AI performed roughly as well as a knowledgeable non-expert human reader, such as a student teacher, but it could not match the performance of a seasoned expert who could instantly spot deep conceptual errors or verify calculations.

The most significant finding was not just that the AI works, but how it works and where it needs help. The researcher discovered that the AI is most reliable when the student's response is linguistically explicit. If a student uses clear words to describe a grouping structure, the AI catches it easily. If the student is vague, the AI gets lost. This suggests that the design of the test questions themselves matters immensely; questions that prompt students to write clearly and fully will yield better results from AI tools. Furthermore, the study concluded that these AI tools should not be seen as replacements for teachers. Instead, they act as a powerful catalyst. They can do the heavy lifting of reading and sorting thousands of responses, flagging the ones that need a human eye. This frees the teacher to focus on the most difficult cases and to make the final instructional decisions based on a clearer, more organized picture of the class's understanding.

The researcher emphasizes that the AI is a tool for the teacher, not a substitute for their judgment. The system is designed to work in a loop where the machine provides a first draft of the analysis, and the human teacher reviews it, correcting any mistakes and adding the necessary context that only a person can provide. This approach respects the complexity of teaching while acknowledging the practical limits of human time. By using AI to handle the initial sorting, teachers can potentially engage with the deep conceptual thinking of every student, rather than just a few. The study confirms that while AI cannot yet replace the nuanced understanding of a master teacher, it has reached a point where it can make the dream of deep, personalized formative assessment in mathematics a practical reality for schools.

Technical Summary: AI-Supported Formative Assessment of Conceptual Understanding in Mathematics

Problem Statement
Formative assessment in mathematics, particularly regarding conceptual understanding, relies heavily on open-ended tasks (e.g., word problem posing, explaining reasoning) to reveal student thinking. However, the practical burden of analyzing these responses prevents teachers from realizing their diagnostic potential; empirical evidence suggests fewer than 15% of such responses are ever scored in digital platforms. While automated scoring has been explored via rule-based systems and supervised machine learning, these approaches often require extensive labeled datasets or fail to capture content-specific conceptual distinctions. Large Language Models (LLMs) offer a potential solution by processing natural language prompts without task-specific fine-tuning. However, a critical gap remains: LLMs process language tokens rather than mathematical meaning. It is an open empirical question whether prompt-based LLMs can reliably identify theory-derived conceptual categories in authentic, linguistically sparse student responses, particularly in domains like multiplication and fractions where surface linguistic signals may not correlate with deep structural understanding.

Methodology
The study utilizes two distinct datasets from the "Mastering Math Online-Check" platform, a digital formative assessment tool used in Grades 5–6 classrooms in Germany. All data is anonymized and compliant with data protection regulations.

Dataset 1 (RQ1): 13,082 codings from 1,763 students across 21 items covering natural numbers and fractions. This dataset was used during the iterative development of the prompt architecture.
Dataset 2 (RQ2): 1,186 responses to a single "problem posing" item (create a word problem for $6 \times 5 = 30$ ). This dataset was collected after prompt stabilization to provide a methodologically cleaner basis for in-depth analysis.

Coding Framework:
Responses were coded against theory-derived categories.

RQ1: Binary coding for three types: Conceptual Understanding (e.g., explicit grouping structure), Misconception (e.g., confusing multiplication with addition), and No Evidence.
RQ2: A two-level approach. First, Holistic Classification into five mutually exclusive categories (ranging from "Correct: multiplicative situation" to "No evidence"). Second, Analytic Rubric Coding of binary subcategories within the holistic frame (e.g., "grouping structure present," "calculation error," "question missing").

LLM Configuration:
Four self-hostable open-source models were evaluated to ensure data privacy compliance: OpenAI GPT OSS 120b, Meta LLaMA 3.1 8B Instruct, Mistral Large, and DeepSeek R1 Distill LLaMA 70B.

Prompt Architecture: A four-component structure was developed iteratively, translating subject-matter expertise into operationalized prompt descriptions. This included category definitions, illustrative student responses, and meta-structure instructions.
Evaluation: LLM performance was benchmarked against "ground truth" established by two subject-matter experts. A subsample (n=255) was also coded by two pre-service teachers (PST1: novice; PST2: experienced) using the same prompts to establish a human baseline.
Metrics: Weighted Precision, Recall, F1-Score, and Accuracy were calculated. Weighted metrics were prioritized due to class imbalance.

Key Results

Overall Performance (RQ1): Across 21 items, the best-performing model (OpenAI GPT OSS 120b) achieved a mean weighted F1-score of 0.818. Performance was higher for Conceptual Understanding (F1=0.863) and Misconception (F1=0.839) categories than for No Evidence (F1=0.625), primarily due to lower recall in identifying the latter.
Holistic vs. Analytic (RQ2):
- Human Baseline: PST2 (experienced) significantly outperformed PST1 (novice) on both holistic (F1 0.892 vs. 0.771) and analytic tasks. PST2's superior performance was attributed to subject-matter expertise enabling arithmetic verification and disambiguation, tasks where PST1 struggled.
- Model vs. Human: The best LLM (OpenAI) matched PST1's holistic performance (F1 ~0.765) but did not reach PST2's level. On the analytic rubric, all models showed a systematic directional bias: they significantly over-assigned subcategories (high recall, lower precision), whereas human raters showed symmetric error distributions.
Error Patterns:
- Reliable Detection: Models reliably identified subcategories where positive features were explicitly present (e.g., "grouping structure present," "explicit grouping language").
- Systematic Failures: Models struggled with subcategories requiring arithmetic verification (e.g., checking if $6 \times 5 = 30$ ) or the recognition of absent features (e.g., missing question). These categories were identified perfectly by PST2 but poorly by both PST1 and all LLMs.
- Ambiguity: Models performed poorly on conceptually ambiguous subcategories where linguistic signals were present but not unambiguous, often resolving them based on token distribution rather than mathematical logic.
Model Variance: Performance varied significantly across models. Meta LLaMA 3.1 8B performed near chance levels, while OpenAI and DeepSeek showed higher, though distinct, error profiles (e.g., DeepSeek under-assigned minor errors; Mistral over-assigned them).

Key Contributions

Empirical Feasibility: The study provides evidence that prompt-based LLMs can achieve high agreement (F1 > 0.8) with expert human codings for theory-derived conceptual categories in authentic classroom settings, without task-specific fine-tuning.
Prompt Architecture: It documents a transferable prompt architecture that operationalizes mathematics-specific pedagogical content knowledge (PCK) for LLMs, demonstrating that detailed, theory-grounded rubrics are essential for performance.
Granular Error Analysis: It identifies specific conditions where automated coding fails: arithmetic verification, detection of absent features, and resolution of linguistic ambiguity. It highlights that these failures mirror the limitations of less experienced human raters, suggesting that subject-matter expertise is the differentiator.

Significance and Claims
The paper argues that LLMs should not be viewed as autonomous assessment agents but as catalysts within the formative assessment cycle.

Human-in-the-Loop: The system is designed to make the analytical work tractable by providing a structured "first-pass" categorization. This redirects teacher attention from reading every response individually to reviewing pre-organized diagnostic information.
Scaffolding for Teachers: For teachers lacking detailed content-specific categories, LLM pre-coding (paired with category descriptions) can scaffold their interpretation, directing attention to conceptually relevant distinctions.
Limitations of Automation: The study explicitly claims that LLMs cannot replace the interpretive authority of the teacher, particularly regarding arithmetic verification and contextual instructional decisions. The "human-in-the-loop" design is presented as a necessity, not just a recommendation, to mitigate systematic biases (e.g., over-estimation of feature presence) and to handle cases requiring deep mathematical reasoning.
Task Design: The findings suggest that "AI-sensitive task design"—tasks that elicit linguistically explicit articulations of conceptual structure—can significantly improve coding reliability.

The paper concludes that while LLM-based pre-coding is feasible and educationally meaningful for a substantial subset of categories, its integration requires careful system design that leverages subject-matter expertise in prompt development and maintains teacher authority in the final interpretive loop.

More like this