Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions

Technical Summary: Using an LLM to Investigate Students' Explanations on Conceptual Physics Questions

Problem Statement
Research-based multiple-choice (MC) assessments, such as the Energy and Momentum Conceptual Survey (EMCS), are standard tools for evaluating conceptual understanding in physics due to their scalability. However, the authors argue that MC formats suffer from significant limitations: they fail to uncover the diverse ways students reason about physics concepts, do not reveal the specific conceptual resources students activate, and may inadvertently reinforce incorrect associations through repeated exposure to distractors. While open-ended written responses offer deeper pedagogical insights, grading them at scale (e.g., in courses with $N > 1,000$) is prohibitively labor-intensive and prone to inconsistencies. The study addresses the need for a scalable, consistent method to assess students' written explanations to identify these underlying conceptual resources.

Methodology
The study was conducted in a first-semester calculus-based physics course at a large US university involving 1,131 students. The researchers administered the EMCS as a pre-test and post-test. For the post-test, three specific questions (Q05, Q16, and Q23) were converted from their original MC format to open-ended written prompts. These questions were selected based on literature indicating low post-test gains and poor performance, suggesting students frequently activate unproductive conceptual resources on these items.

Students were instructed to respond using the Claim-Evidence-Reasoning (CER) framework, a method they had practiced throughout the semester. The data analysis utilized OpenAI's GPT-4o Large Language Model (LLM). The LLM was prompted via a structured template acting as an "experienced physics educator" with a specific rubric to:

1. Evaluate the correctness of the student's claim (answer).
2. Evaluate the correctness of the evidence and reasoning.
3. Assign emergent categories to incorrect explanations based on distinct conceptual patterns (up to six categories).
4. Utilize "chain of thought" prompting to step through the physics principles before scoring.

To validate the LLM's performance, a human grader (HG) independently analyzed a random 20% sample ($N=231$) of the responses. The study measured agreement using Cohen's kappa and compared the percentage of correct classifications. The LLM was also tested for consistency across multiple runs with a temperature setting of 0.5.

Key Results

• Grading Accuracy: The LLM demonstrated high agreement with human graders. The discrepancy in overall correctness (claim supported by correct reasoning) was remarkably low, ranging from 0% to 3% across the three questions. The mean Cohen's kappa between the LLM and the human grader was 0.77 (substantial to almost perfect agreement), with individual question scores of 0.74, 0.65, and 0.89.
• Emergent Categories: The LLM successfully generated emergent categories for incorrect explanations that were not predefined. For example, on Q05, the LLM identified categories such as "Misapplying Collision Type" and "Misidentifying Collision Type," which were not present in the original MC distractors.
• Revealing Hidden Misconceptions: The analysis revealed instances where students provided the correct claim (answer) based on incorrect or unproductive reasoning. In the MC format, these students would have been marked as correct; however, the LLM identified the flawed conceptual resources (e.g., misapplying conservation of momentum in an inelastic collision context) that led to the correct answer by chance or flawed logic.
• Consistency: The LLM produced consistent emergent categories across multiple runs, with low standard deviation in the percentage of students marked as fully correct.

Significance and Claims
The authors claim two primary contributions:

1. Reliability: Appropriately prompted LLMs (specifically GPT-4o) can reliably assess the overall correctness of students' written responses, including both the answer and the supporting explanation, with agreement levels comparable to human graders. This suggests LLMs can alleviate the bottleneck of grading open-ended physics questions at scale.
2. Depth of Insight: LLM-based analysis reveals emergent categories of incorrect reasoning that are not captured by the fixed options of traditional MC assessments. The study demonstrates that LLMs can uncover "myriad ways of thinking" and specific conceptual resources (both productive and unproductive) that remain hidden in standard MC formats.

The paper concludes that while MC assessments are sufficient for quantifying the percentage of students with correct conceptions, LLMs offer a promising avenue for educators to isolate specific misunderstandings and investigate the richness of student reasoning in large-enrollment courses. The authors note a limitation regarding the subset of questions used (due to concerns about student fatigue) and state that future work will aim to expand this analysis to the full EMCS and more complex physics assessments.