A Tool-Invariant Framework for Teaching and Assessing Computational Methods in the Age of Agentic AI

The New Math Class: When the Calculator Writes the Code

Imagine you are learning to cook. For centuries, the hardest part of cooking was chopping the vegetables and stirring the pot. You had to learn the knife skills, the timing, and the muscle memory to make a perfect sauce. But then, someone invented a robot chef. This robot doesn't just chop; it can read a recipe you describe in plain English, chop the onions, stir the sauce, taste it, and even fix it if it's too salty. Suddenly, the act of "cooking" changes. You don't need to be a master knife-wielder anymore; you need to be a master taster and a clear instruction-giver. If you just say "make it tasty," the robot might make a disaster that looks delicious.

This is exactly what is happening in the world of physics and science education. For a long time, learning "computational physics" meant learning to write computer code—typing out instructions for a computer to solve complex math problems, like predicting how a planet moves or how a virus spreads. But a new kind of artificial intelligence (AI) has arrived. These aren't just chatbots that answer questions; they are "agentic" tools. They can take a simple sentence like "simulate a bouncing ball with air resistance" and write the code, run the simulation, draw the graphs, and fix their own mistakes.

The big question for teachers is: If the robot does the cooking, what do the students need to learn? Do they still need to know how to chop? And if the robot hands in a perfect-looking report, how do we know the student actually understands the recipe? This paper tackles that exact problem, arguing that the old way of grading (just looking at the final report) is broken, and proposing a new way to teach and test students that focuses on understanding the why and how, rather than just the typing.

---

The Paper's Big Idea: From Typing to Thinking

The author, Larry Engelhardt, argues that we are going through a massive shift, similar to when calculators replaced slide rules. In the past, students had to learn the "mechanics" of the tool: the specific commands, the syntax, and the typing skills to make the computer work. Now, with these new AI agents, the "mechanics" are handled by the machine. The student's job has shifted from being the operator to being the director.

The paper suggests that while the tool changes, the core things a student must know have stayed the same for centuries. Engelhardt organizes these into five "pillars" that remain true whether you are using a pencil, a calculator, or a super-smart AI:

1. Inputs and Outputs: Knowing what you are putting into the system and what you expect to get out.
2. Conceptual Understanding: Knowing what the method actually does and how it works, not just how to type it.
3. Terminology: Using the right scientific words to describe the problem clearly.
4. Sensemaking: The ability to look at a result and say, "That makes sense," or "That looks wrong."
5. Operating the Tool: The skill of using the current tool (which used to be typing, but is now giving clear instructions to an AI).

The paper's main finding is that in the age of AI, the most important skill has become Sensemaking (Pillar 4). Because the AI can generate code and graphs instantly, the student's ability to verify if those results are real is now the most critical skill. The author calls this "verification authority." Just because the AI says "here is the answer" doesn't mean it's right; the student must be the judge.

Why the Old Tests Are Broken

The paper explicitly argues against the traditional way of grading computational classes. For decades, teachers would assign a project: "Write code to simulate a pendulum, make a graph, and submit a report." The assumption was that if the student submitted a working report, they must understand the physics.

Engelhardt says this is no longer true. He calls this the "proxy collapse." A "proxy" is a stand-in for something else. The report was a stand-in for the student's understanding. But now, because an AI can generate a perfect report on demand, the report no longer proves the student knows anything. It's like if a student handed in a perfect essay written by a famous author; the essay is good, but it doesn't prove the student can write. The paper argues that we must stop treating the "artifact" (the code or the graph) as the proof of learning.

The New Solution: The "Oral Defense"

So, if we can't trust the homework, how do we test students? The paper proposes a practical, two-part system designed for the small classes where physics is usually taught:

1. AI-Free Quizzes: Students still take short, in-class coding quizzes where they cannot use AI. This ensures they haven't lost the basic "muscle memory" of how the code works. They might be asked to fix a broken piece of code or write a small function from scratch.
2. The Oral Defense: This is the big new idea. After students use AI to help them build a project, they must sit down with the teacher for a 10-minute conversation.
   • The Twist: Before the meeting, the teacher uses a tool to strip all the comments (the notes explaining the code) from the student's submission.
   • The Test: The teacher asks the student to walk through their own code, explain what the graphs mean, and answer questions like, "Why did you choose this method?" or "What would happen if we doubled the mass?"
   • The Gatekeeper: There is a specific "verification gate." The student must prove they checked their work. If they can't explain why the result is correct, they fail, even if the code runs perfectly.

The author suggests that this oral defense is the only way to certify that the student actually owns the work. It's similar to how a Ph.D. student defends their thesis; the paper isn't just graded on the writing, but on the student's ability to explain and defend it in person.

What the Paper Says (and Doesn't Say)

The paper is careful to note that this is a new framework based on the author's experience and adaptation, not a result of massive, long-term scientific studies. The author admits that we don't yet know the perfect "dose" of coding practice needed before switching to AI. They suggest that students still need to see the "white box" (the code working transparently) at least once to build a mental model, but they aren't sure exactly how much is enough.

The paper also rules out the idea that we can just "ban" AI. The author argues that trying to stop students from using AI at home is like trying to stop them from using a calculator; it's impossible. Instead, the focus must be on motivation and assessment design that makes using AI the right way to learn, not the easy way to cheat.

The Bottom Line

The paper concludes that the "product" of a computational physics course is no longer the code or the graph. Those are cheap and easy to get now. The real product is the student's ability to explain and defend the results using the language of physics.

In a world where machines can do the heavy lifting, the human value lies in the judgment. The paper suggests that the future of teaching isn't about stopping the robot from writing the code; it's about teaching the student how to be the smart, critical director who knows exactly what the robot is doing, why it's doing it, and whether the answer is actually true. The conversation between the student and the teacher becomes the most important part of the class, because that's the only place where true understanding can be proven.