Reliable isomorphic physics problem generation with large language models
This study introduces an AI-powered system utilizing large language model agents to reliably generate isomorphic physics problems that preserve core conceptual structures while varying superficial details, achieving an 89% success rate in producing directly usable questions for introductory mechanics courses.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are a teacher trying to create a math or science test. You have a great question about a falling apple, but you need a dozen different versions so students can't just copy each other's answers. You want the new questions to feel fresh—maybe about a falling pineapple or a sliding skateboard—but they must test the exact same brain muscles and physics rules as the original. This is the tricky art of making "isomorphic" problems: puzzles that look different on the outside but are identical twins on the inside. For decades, computers have been good at just swapping numbers (turning a 5-meter drop into a 10-meter drop), but they have struggled to rewrite the whole story without breaking the logic. Now, a new wave of super-smart computer programs called Large Language Models (LLMs) is trying to learn this art. These models are like digital scribes that have read almost everything ever written, but they are notorious for sometimes making things up or getting the logic wrong. The big question for educators is: Can we trust these digital scribes to write brand-new, perfect test questions that a teacher could use immediately, or do they still need a human to fix their mistakes?
This paper introduces a clever new system designed to answer that question by building a "robot factory" for physics problems. The researchers, Xian Wu and Lindim Ismaili from the University of Connecticut, didn't just ask a single AI to "make a new question." Instead, they built a team of specialized AI agents that work together like a well-oiled assembly line. First, one agent reads the original problem and breaks it down into its core parts. Another agent checks the math to make sure the answer is actually right. A third agent rewrites the story, swapping the apple for a rocket or the numbers for new values, while carefully keeping the underlying physics rules exactly the same. Finally, a fourth agent formats everything into a clean, printable document. They even hosted this whole factory on a tiny, affordable computer called a Raspberry Pi, making the tool available to the public.
To see if this factory actually works, the team tested it with 13 real physics questions from a college-level mechanics course. They ran the system 20 times, asking it to generate 13 new versions for each run. The results were surprisingly promising: out of all the questions the system produced, 89% were rated as "fully specified and directly usable." This means that for nearly nine out of ten questions, a teacher could pick them up and use them on a test without needing to rewrite a single word or fix a calculation error. The system successfully kept the "same" physics concepts while changing the "different" story details, just as the researchers hoped.
However, the paper is careful not to call this a perfect solution. While the system showed significant potential, it also hit some walls. The researchers noted that the system isn't flawless; it still produces errors occasionally, and the process relies on the AI understanding complex human logic well enough to avoid "hallucinations" (where the AI invents fake facts). The study suggests that while these AI tools are getting very good at generating instructional content, they are not yet ready to completely replace human experts. Instead, the paper argues that the best path forward is to use these AI systems as powerful assistants that handle the heavy lifting of rewriting, while human teachers provide the final oversight to ensure everything is perfect. The future of physics education might not be a robot taking over the classroom, but a robot helping teachers create endless, fresh challenges for their students.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.