🔬 physics

Probing AI-generated physics solutions and preparing students to critique them

This study demonstrates that well-specified prompts yield more complete AI-generated physics solutions while guiding students to use the MAPS rubric for reflection significantly enhances their ability to critically identify reasoning flaws and errors in AI-generated answers compared to solving problems independently.

Original authors: Nikhil Sanjay Borse, Amir Bralin, Sean Savage, N. Sanjay Rebello

Published 2026-08-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikhil Sanjay Borse, Amir Bralin, Sean Savage, N. Sanjay Rebello

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of physics as a giant, intricate puzzle where the pieces are forces, motion, and energy. For decades, students have learned to solve these puzzles by following strict rules, drawing diagrams, and doing math step-by-step. But recently, a new kind of "super-tutor" has entered the classroom: Artificial Intelligence (AI). These AI models are like incredibly fast, well-read robots that can read a physics problem and spit out a solution in seconds. They sound amazing, right? But here's the catch: just because an answer looks neat and sounds confident doesn't mean it's actually correct. Sometimes, these AI tutors skip steps, make up facts, or get confused by pictures. This creates a tricky situation for students. If the robot tutor is sometimes wrong, how do students learn to spot the mistakes before they copy them? This is the big question scientists are asking: How do we teach students to be the "editors" of AI, rather than just the copy-pasters?

This paper dives into that exact problem by treating a physics problem like a game of "telephone" with a robot. The researchers wanted to see two things: first, how changing the way you ask a question (the "prompt") changes the robot's answer, and second, how to best train students to catch the robot's errors. They used a specific type of physics problem involving a ball rolling inside a spinning bowl—a scenario that requires both math and a good mental picture of how things move.

First, they tested the AI, which they called "o4-mini," with three different types of instructions. Think of it like ordering food. In the first scenario, they gave the AI a "well-specified" order: "Here is the ball, here is the bowl, here is the speed, and here is exactly what to calculate." The AI did a great job, serving up a complete, correct meal. But when they gave it a "vague" order—leaving out details and hoping the AI would guess the rest—the AI started to hallucinate. It made up assumptions, messed up the math, and served a dish that looked okay but tasted wrong. Finally, they tried a "multimodal" order, giving the AI a picture of the bowl along with the text. Even with the picture, the AI got confused; it described the food but forgot to actually cook the numbers, skipping the hard math entirely. The study found that the more details you leave out, or the more you rely on pictures without clear text, the more likely the AI is to make a mistake that sounds plausible but is actually broken.

Then, the researchers brought in 24 groups of college students to play the role of the critics. They split the students into two teams to see how to best prepare them for the job. The first team was the "Doers." They were asked to solve a similar physics problem on their own, without any help, to get their brains warmed up. The second team was the "Critics." Instead of solving a problem, they were given a checklist based on a famous grading rubric called MAPS (which stands for Minnesota Assessment of Problem Solving). This checklist asked them to look for specific things like "Did they draw a diagram?" "Did they define their variables?" and "Did they skip a math step?"

After their warm-up, both teams were handed the same messy, AI-generated solution to the spinning bowl problem and asked to grade it. The results were fascinating. The "Doers" team, who had just solved a problem themselves, often fell for the AI's smooth writing. They gave the AI high scores because the answer looked organized and confident, or they criticized it based on their own misunderstandings of physics. They were like students who think a long essay must be good just because it has big words.

However, the "Critics" team, who had practiced using the MAPS checklist, were much sharper. They didn't get fooled by the fancy language. Instead, they pointed out the specific flaws the researchers had found earlier: the AI had skipped the hard math, used confusing symbols without explaining them, and failed to draw a free-body diagram. They were like professional editors who know exactly where to look for typos, regardless of how pretty the font is.

The paper suggests that while solving problems on your own is good for learning physics, it doesn't automatically teach you how to spot AI errors. To become a true AI critic, students need to practice looking at solutions with a specific set of eyes—checking for missing steps, undefined terms, and logical gaps. The study concludes that as AI gets better at sounding smart, students will need these structured tools to separate the real physics from the robot's confident nonsense. It's not about trusting the robot or hating it; it's about learning to be the boss who knows exactly what a correct answer should look like.

Technical Summary: Probing AI-generated physics solutions and preparing students to critique them

Problem Statement
Large Reasoning Models (LRMs), such as OpenAI's o4-mini, demonstrate promise in solving multi-step physics problems. However, existing benchmarks often rely on fixed, fully specified prompts, leaving a gap in understanding how variations in prompt design—specifically prompt-specificity and modality—affect model reasoning and correctness. Furthermore, while AI can generate fluent and plausible physics solutions containing conceptual errors or incomplete reasoning, there is limited research on how to effectively prepare students to critically evaluate these AI-generated outputs. This study addresses two interconnected questions: how prompt variations influence the quality of LRM physics solutions, and whether specific preparatory strategies enable students to identify and critique weaknesses in AI-generated reasoning more effectively than independent problem-solving alone.

Methodology
The study employed a two-phase experimental design grounded in Jonassen's problem classification framework and the Minnesota Assessment of Problem Solving (MAPS) rubric.

Phase I: Model Evaluation via Prompt Variation
The researchers utilized a rotational-mechanics problem involving a sphere rolling inside a hollow rotating hemisphere. They tested OpenAI's o4-mini model using three distinct prompt variations:
1. Well-specified (Text-only): A prompt with all necessary details explicitly provided.
2. Underspecified (Text-only): A prompt where key details were removed, requiring the model to infer assumptions.
3. Multimodal: A prompt combining text with an AI-generated diagram.
Each prompt was submitted five times. Outputs were evaluated for quantitative accuracy and scored using the MAPS rubric across five dimensions: Useful Descriptions, Physics Approach, Specific Application, Mathematical Procedures, and Logical Progression. Two expert raters independently scored the outputs to establish reliability.
Phase II: Student Critique Preparation
The study involved 24 introductory physics lab groups (72 students) at a large U.S. Midwestern university. Groups were assigned to one of two preparation conditions before evaluating a complex, multimodal AI-generated solution to the rotational-mechanics problem:
1. Solution Condition: Groups independently solved a related, simpler rolling-sphere problem without AI assistance.
2. Critique Condition: Groups critiqued an AI-generated solution to the same simpler problem using 15 reflection questions structured around the five MAPS categories.
Following preparation, all groups evaluated the complex AI solution, providing MAPS-aligned scores, written justifications, and survey responses regarding perceived accuracy. Data were analyzed qualitatively to identify patterns in critique quality and justification.

Key Results

Impact of Prompt Design on LRM Performance:
- Well-specified prompts yielded the most complete solutions, with the model achieving correct quantitative results and high scores across all MAPS categories, except for minor gaps in explaining mathematical procedures.
- Underspecified prompts led to significant reasoning failures. While the model identified the physical structure, it neglected key conservation laws (rotational kinetic energy, angular momentum) and produced incorrect numerical results due to flawed assumptions.
- Multimodal prompts resulted in a qualitative rather than quantitative response. The model skipped the necessary numerical integration, providing a verbal description and a qualitative plot instead of a calculated height evolution. It also failed to define variables clearly or include necessary free-body diagrams.
- Conclusion: Prompt specificity and modality directly shaped reasoning quality. Underspecification increased the likelihood of missing assumptions and numerical errors, while multimodal inputs led to the omission of quantitative procedures.
Impact of Preparation on Student Critique:
- Solution Condition: Students who solved a related problem independently often accepted the AI solution uncritically due to its polished appearance or critiqued it based on their own misconceptions rather than expert-aligned criteria.
- Critique Condition: Students who engaged in MAPS-guided reflection demonstrated more expert-aligned critiques. They were more likely to identify specific weaknesses such as missing numerical integration, undefined notation (e.g., the rolling angle $\Psi$ ), skipped computational steps, and the mismatch between qualitative plots and quantitative requirements.
- Perceived Accuracy: While both groups expressed general caution about AI, the Critique group's skepticism was more often grounded in specific, defensible features of the solution (e.g., lack of integration, unclear notation), whereas the Solution group's skepticism was sometimes based on traditional misconceptions or a lack of confidence in their own ability to verify the work.

Key Contributions

Systematic Prompt Analysis: The study demonstrates that LRM performance in physics is not static but highly sensitive to prompt specificity and modality. It provides empirical evidence that removing details or adding visual representations can degrade the model's ability to perform necessary quantitative reasoning.
Evaluation Framework: The research validates the use of the MAPS rubric not only for assessing student work but also for diagnosing specific weaknesses in AI-generated reasoning, such as skipped mathematical procedures and undefined variables.
Pedagogical Strategy: The findings suggest that while solving related problems builds physics knowledge, it does not automatically equip students to critique AI reasoning. Structured reflection using disciplinary criteria (MAPS) is more effective in training students to identify recurring AI weaknesses, such as the conflation of qualitative and quantitative representations.

Significance and Claims
The paper claims to contribute to Physics Education Research (PER) by establishing a dual approach: using prompt variation to map the "recurring weaknesses" of AI reasoning and using those findings to design instructional activities that foster calibrated, evidence-based student critique. The authors argue that as AI outputs become more fluent, surface-level evaluation will be insufficient. Therefore, students must be explicitly trained to evaluate the process of AI reasoning—checking for logical coherence, defined variables, and complete procedures—rather than relying on the apparent coherence of the final answer. The study positions the MAPS rubric as a critical tool for shifting student engagement from passive acceptance to active, disciplinary evaluation of AI-generated content.

Limitations
The authors note that the study is limited to a single LRM (o4-mini) and a specific rotational-mechanics problem family, which may affect generalizability across different models or physics domains. Additionally, the student analysis is based on a relatively small sample (24 groups) and relies on group-level artifacts rather than individual reasoning traces. The authors view the student component as an exploratory comparison of critique patterns rather than a definitive conclusion on instructional efficacy.

More like this