🔬 physics

Assessing AI in Introductory Physics Problem Solving

This study evaluates OpenAI's o4-mini model on introductory physics problems from Halliday and Resnick, finding that while it achieves high overall accuracy (~90%), its performance is significantly constrained by problem modality (dropping from 96% on text-only to 79% on text-image tasks) and increases in difficulty.

Original authors: Amir Bralin, N. Sanjay Rebello

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Amir Bralin, N. Sanjay Rebello

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where your homework helper isn't just a search engine that finds answers, but a brain that actually tries to solve the puzzle for you. This is the realm of Artificial Intelligence, or AI, specifically a type called Large Language Models. Think of these models as super-advanced digital parrots that have read almost every book on the internet. They are incredibly good at mimicking human conversation and spotting patterns in text, like a detective who has memorized every mystery novel ever written. But there's a catch: while they can chat like a professor, they sometimes struggle to "see" the world the way humans do. They might read a description of a falling apple perfectly, but if you show them a picture of a falling apple and ask them to calculate its speed, they might get confused. This is a big deal for science education because physics isn't just about words; it's about understanding how the universe works, often through diagrams, graphs, and tricky problems. As these AI tools get smarter, teachers and students are asking: Can a robot really do my physics homework, or will it just guess until it gets lucky?

In a recent study, researchers put a cutting-edge AI model, called "o4-mini," to the test to see if it could handle the classic physics problems found in the standard undergraduate textbook, Fundamentals of Physics by Halliday and Resnick. They treated the AI like a student taking a final exam, feeding it 1,203 different problems ranging from easy to hard. The results were a mix of "wow" and "whoops." The AI was a star when the problems were just text, solving about 96% of them correctly. It was like a brilliant tutor who could explain concepts perfectly if you just asked in words. However, the moment the problems included images—like diagrams of pulleys, graphs of motion, or sketches of circuits—the AI's performance dropped significantly to about 79%. It's as if the robot suddenly forgot how to read a map when it was handed a picture instead of a list of directions.

The study also discovered that the AI's confidence and accuracy didn't stay steady as the problems got tougher. When the questions were "Easy," the model was a champ. But as the difficulty jumped to "Medium" and then "Hard," its success rate slowly faded, dropping to 84% on the hardest challenges. Interestingly, the AI didn't just give up; it actually tried harder. When faced with a difficult problem, the model generated longer, more detailed answers, using more "tokens" (which are like the digital building blocks of its thoughts) to figure things out. Yet, despite this extra effort, it still made more mistakes on the complex stuff. The researchers found that the AI was surprisingly consistent across different topics, whether it was dealing with gravity, electricity, or quantum mechanics; it didn't seem to have a specific "weak subject" other than the difficulty level and the presence of images.

Ultimately, this paper suggests that while AI is becoming a powerful tool for solving standard physics problems, it isn't quite the perfect, all-knowing tutor yet. It excels at text-based reasoning but stumbles when it needs to coordinate text with visual information, and its accuracy wavers as problems become more complex. The authors conclude that for now, AI is a helpful assistant that can handle the basics and the middle ground, but it still needs human guidance, especially when the problems get tough or involve pictures. It's a promising step forward, but the robot still has a lot of learning to do before it can truly replace a human physicist in the classroom.

Technical Summary: Assessing AI in Introductory Physics Problem Solving

Problem Statement
While Large Language Models (LLMs) have demonstrated significant advancements in general problem-solving, their specific capabilities and limitations within the domain of introductory physics remain under-explored, particularly regarding the interplay between problem modality (text vs. image) and difficulty levels. Previous studies in Physics Education Research (PER) have highlighted inconsistencies in AI performance on standardized assessments, noting strong text-based reasoning but significant struggles with visual interpretation, graph extraction, and complex multi-step calculations. This study seeks to rigorously evaluate the problem-solving capability of a state-of-the-art reasoning model on a comprehensive set of standard undergraduate physics problems, analyzing performance across different modalities and difficulty tiers.

Methodology
The researchers evaluated OpenAI's "o4-mini" model, a reasoning-scaling architecture designed to "think for longer" before responding. The evaluation dataset consisted of 1,203 end-of-chapter problems from Halliday and Resnick's Fundamentals of Physics (Volumes 1 and 2), covering mechanics, thermodynamics, electromagnetism, optics, relativity, and quantum theory.

The data processing pipeline involved:

Data Extraction: Manually converting textbook problems into a structured format. Text and numerical data were preserved, while equations and symbols were re-typed using the siunitx package for typographical consistency.
Modality Handling: Images (figures and tables) were extracted as screenshots at fixed resolutions to preserve relative scaling. Problems were categorized as "Text-only" (TXT) or "Text + Image" (IMG).
Execution: Each problem was presented to the o4-mini model via the OpenAI API. To account for the stochastic nature of LLMs, each problem was solved $n=5$ times independently.
Scoring: A separate, more powerful model (GPT-5) was employed to grade the generated solutions against the textbook's answer key. The scoring process was validated on a sample of 600 solutions, yielding a 97.67% agreement rate with human expectations, establishing a systematic error margin of 0.023.
Statistical Analysis: Accuracy was defined as the mean score across runs. The study utilized logistic regression and generalized estimating equations to analyze the relationship between accuracy and problem difficulty (Easy, Medium, Hard), treating difficulty as both an ordinal variable and distinct categories.

Key Results
The study yielded three primary findings regarding the performance of the o4-mini model:

Overall Accuracy: The model achieved an overall accuracy of 90% ± 3% on the standard introductory physics curriculum.
Modality Dependence: Performance was heavily dependent on the input modality. The model solved text-only problems with 96% ± 3% accuracy. However, accuracy dropped significantly to 79% ± 4% for problems requiring the coordinated interpretation of text and images.
Difficulty Dependence: A clear inverse relationship was observed between problem difficulty and accuracy. Accuracy declined from 94% (Easy) to 88% (Medium) and further to 84% (Hard).
Computational Effort: Contrary to the trend in accuracy, the "effort" exerted by the model (measured in output tokens) increased with difficulty. The model generated nearly double the token output for Medium and Hard problems compared to Easy ones, yet this increased computational cost did not compensate for the drop in accuracy.
Topic Uniformity: Unlike previous findings suggesting specific conceptual weaknesses, the model's performance remained relatively uniform across the six major physics topics (Mechanics, Waves, Thermodynamics, Electromagnetism, Optics, and Quantum Theory), with no distinct patterns of failure tied to specific subject matter.

Significance and Claims
The paper claims that while state-of-the-art LLMs are capable of solving the majority of standard introductory physics problems with high accuracy, their utility is currently constrained by two main factors: modality and difficulty.

Modality Limitation: The study reinforces that the inability to effectively process visual inputs remains a central weakness. In physics, where diagrams and graphs are often essential for problem formulation, this limitation prevents the model from acting as a complete problem-solving agent.
Difficulty Scaling: The observed decline in accuracy as problems become more complex suggests that while models excel at pattern recognition and straightforward applications of principles, they struggle to integrate multiple layers of physical reasoning required for complex phenomena.
Objective Difficulty Metric: The authors propose that the computational effort (token count) required to solve a problem, coupled with the success rate, could serve as an objective, computationally derived metric for problem difficulty, potentially aiding in the classification of test banks.
Educational Implications: The uniform performance across topics suggests that these models do not suffer from topic-specific conceptual gaps in the same way novice students might. However, the reliance on text-only success implies that for educational use, AI is currently a reliable tool for checking reasoning in text-based scenarios but remains unreliable for problems requiring visual synthesis.

The authors conclude that while AI models are approaching expert-level performance in text-based physics problem solving, the gap in visual interpretation and complex reasoning means they are not yet ready to replace human problem solvers in a fully integrated educational or scientific context.

More like this