🔬 physics

Using LLMs to Detect Growth in Computational Thinking in Introductory Physics

This study demonstrates that Large Language Models can effectively scale the assessment of student growth in computational thinking within introductory physics courses by mirroring human evaluations of written explanations for data practices and problem-solving, though both methods face challenges in reliably evaluating complex constructs like systems thinking.

Original authors: Sean Savage, Anand Shanker, Grace Michlitsch, N. Sanjay Rebello

Published 2026-08-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sean Savage, Anand Shanker, Grace Michlitsch, N. Sanjay Rebello

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of physics not just as a collection of formulas on a chalkboard, but as a massive, interactive video game where the laws of nature are the code. For decades, teachers have tried to get students to stop just memorizing the rules and start writing the code themselves. This skill is called "Computational Thinking." It's not just about typing fast or knowing computer syntax; it's about breaking a giant, messy real-world problem (like a car crash or a planet's orbit) into tiny, logical steps a computer can understand. It's like being a translator who speaks both "Human" and "Machine," figuring out how to describe a falling apple so a robot can predict exactly where it will land.

The big question for educators is: How do we know if students are actually getting better at this translation job? Traditionally, teachers have used multiple-choice tests, which are easy to grade but terrible at showing how a student thinks. To see the real thinking, students have to write out their answers in words. But here's the catch: reading hundreds of handwritten essays is exhausting, slow, and expensive. It's like trying to grade a million essays by hand while the clock is ticking. Now, a new kind of "super-reader" has arrived: Artificial Intelligence (AI), specifically Large Language Models (LLMs). These are the same types of smart computers that can write stories or chat with you. The burning question is: Can these AI bots read student essays and spot growth in their thinking skills as well as a human teacher can, but without the coffee breaks?

This paper, written by researchers at Purdue University, sets out to answer that question. They wanted to see if an AI could act as a giant, tireless teaching assistant to track how students' computational thinking grows over a semester. They didn't just ask the AI to grade; they asked it to look for specific "superpowers" in the students' writing: Can they handle data? Can they break problems down? Can they understand how complex systems work together?

To test this, the researchers gathered 936 students in a physics class where they learned to code in Python. Before the class started and after it ended, the students answered open-ended questions about physics problems, like interpreting a graph of a bouncing spring or designing a simulation for a sled sliding down a hill. First, a team of human experts read a small sample of these essays and gave them scores based on a strict rubric. This created a "gold standard" of what a human thinks is a good answer. Then, they fed the exact same essays to an AI model (specifically a multimodal version of GPT-5.4-mini) and asked it to grade them using the same rules.

The results were surprisingly promising. The AI turned out to be a very capable assistant. For clear-cut skills like handling data (Data Practices) and breaking down code steps (Computational Problem-Solving), the AI's grades matched the human experts' grades almost perfectly. When the researchers looked at the whole group of 936 students, the AI successfully spotted the same big trends the humans found: students got significantly better at handling data and solving computational problems by the end of the semester. In fact, the AI confirmed that students made huge leaps in these areas, with the data skills showing a massive improvement.

However, the story isn't a perfect "AI wins everything" tale. The researchers found that both humans and the AI struggled with the most complex skill: "Systems Thinking." This is the ability to see how all the tiny parts of a system interact to create a big picture. Because this concept is so fuzzy and hard to pin down, even the human experts disagreed with each other sometimes, and the AI struggled to match them too. The paper suggests that the AI isn't "dumb" here; rather, the task itself is just really hard to define clearly.

There was also a funny twist with one specific question about designing a sled simulation. The students were already so good at this topic before the class even started that they couldn't get much better. It was like asking a professional chef to improve their knife skills when they were already using the sharpest knives in the world. The scores hit a "ceiling," meaning there was no room for growth to be measured, and the AI correctly identified that no progress happened there.

In the end, the study shows that AI is a viable, powerful tool for checking how students learn to think like computational physicists. It can scale up to read thousands of essays and find the same patterns humans do, saving teachers from drowning in paperwork. But it also serves as a reminder that AI is only as good as the rules we give it. For the messy, complex parts of thinking where even humans disagree, we still need careful design and human oversight. The AI isn't replacing the teacher; it's handing them a super-powered magnifying glass to see the growth that was already happening.

Technical Summary: Using LLMs to Detect Growth in Computational Thinking in Introductory Physics

Problem Statement
As computation becomes a "third pillar" of physics education alongside experiment and theory, educators face the challenge of fostering authentic Computational Thinking (CT) rather than superficial programming skills. While open-ended written responses offer a rich window into student reasoning and the integration of physical laws with code, evaluating them at scale is resource-intensive and often suffers from inter-rater reliability issues. Conversely, multiple-choice assessments lack the depth to capture mental models. The increasing availability of generative AI further complicates assessment by allowing students to bypass cognitive effort. This study addresses the need for a scalable, reliable method to evaluate complex, open-ended CT assessments in large-enrollment physics courses, specifically investigating whether Large Language Models (LLMs) can effectively mirror human evaluation of student growth in CT practices.

Methodology
The study utilized a pre-test/post-test design with 936 students in a calculus-based engineering physics course at a large Midwestern university. The course integrated computation heavily, requiring weekly Python-based labs.

Instrument Development: Three open-ended survey questions were developed based on the Weintrop et al. taxonomy of CT, covering four domains: Data Practices (DP), Computational Problem-Solving Practices (CPP), Modeling and Simulation Practices (MSP), and Systems Thinking Practices (STP).
- Q25: Interpreted a Python-generated energy vs. position graph (DP, CPP, STP).
- Q26: Traced algorithmic logic in a Python script for elastic collision (CPP, STP).
- Q27: Proposed parameters for a sled simulation (MSP, STP).
Human Baseline: A specialized rubric was developed to score responses on a 0–2 scale for each CT practice and Physics Correctness. Three researchers independently coded a subset of 50 students (300 responses) to establish a "ground truth" via majority vote after achieving consensus on operational definitions. Inter-rater reliability (Fleiss' $\kappa$ ) was calculated.
LLM Validation: A multimodal LLM (GPT-5.4-mini) was deployed via API. The model received the survey image, prompt text, and student response. Using a structured prompt that assigned the model the role of an expert physics education researcher and provided the exact human rubric, the LLM generated justifications and scores. To ensure consistency, the temperature was set to 0.2, and each response was evaluated across three independent runs, with the final score determined by majority vote.
Analysis: The LLM's scores on the 50-student subset were compared against the human baseline to calculate agreement. Subsequently, the validated LLM pipeline was applied to the full dataset ( $N=936$ ) to detect growth trends.

Key Results

Human Reliability: Human coders achieved substantial agreement on explicit tasks (MSP $\kappa=0.80$ , DP $\kappa=0.80$ ) and Physics Correctness ( $\kappa=0.73$ ). Agreement was lower for the more complex Systems Thinking (STP) construct ( $\kappa=0.51$ ), reflecting its inherent cognitive demand.
Student Growth (Human-Graded): Analysis of the human-graded subset revealed significant growth in Data Practices ( $d=1.03$ ), Computational Problem-Solving ( $d=0.52$ ), and Systems Thinking ( $d=0.45$ ). Physics Correctness showed small-to-moderate growth ( $d=0.31$ ). No significant growth was observed in Modeling and Simulation Practices ( $d=0.00$ ), attributed to a "ceiling effect" where students entered the course with sufficient prior knowledge to score highly on the initial prompt.
LLM Performance: The LLM demonstrated substantial agreement with human raters on well-defined practices (DP $\kappa=0.90$ , MSP $\kappa=0.78$ , CPP $\kappa=0.69$ ). Agreement was lower for complex, multi-component constructs (Physics Correctness $\kappa=0.53$ , STP $\kappa=0.48$ ), mirroring the lower reliability observed among human coders.
Scalability: When applied to the full dataset, the LLM successfully reproduced the macroscopic trends identified by human raters, detecting significant growth in DP and CPP across Q25 and Q26, and confirming the lack of growth in MSP on Q27.

Key Contributions

Validation of LLMs for CT Assessment: The study demonstrates that LLMs can serve as a viable, scalable mechanism for evaluating open-ended CT assessments, achieving agreement levels comparable to human experts, particularly for clearly defined practices.
Empirical Evidence of CT Growth: The research provides quantitative evidence of student growth in specific CT domains (DP, CPP, STP) within a computationally intensive physics curriculum, while highlighting the difficulty of measuring growth in areas where students possess high prior knowledge.
Identification of Assessment Limits: The study identifies that both human and AI raters struggle with complex, integrated constructs like Systems Thinking, suggesting that current rubrics and prompts may need refinement to distinguish between identifying system components and explaining their interactions.

Significance and Claims
The authors claim that LLMs offer a practical solution to the scalability problem in qualitative physics education research. The study posits that LLMs do not merely automate grading but can reliably detect key pedagogical trends (such as growth in data practices) across large populations. However, the authors remain modest regarding the evaluation of highly integrated constructs; they conclude that while LLMs perform comparably to human experts, the variance in scoring for complex constructs like STP stems from the inherent difficulty of the construct itself rather than a deficiency in the model. Consequently, the paper argues that LLMs are effective for scalable formative assessment but require careful rubric design and human oversight when evaluating complex reasoning. The study does not claim that LLMs have solved the problem of assessing complex reasoning, but rather that they provide a viable path forward for scaling the evaluation of student CT in large-enrollment courses.

Technical Summary: Using LLMs to Detect Growth in Computational Thinking in Introductory Physics

More like this