When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems

This study demonstrates that scalable, real-time AI-generated physics practice can be effectively validated using a streamlined set of automated structural and learner-visible checks, rather than exhaustive scoring, to ensure both technical soundness and student engagement.

The Gist

Imagine you are a student studying for a big physics exam. You've done all the homework, but you still feel shaky on a specific topic. You turn to an AI chatbot and say, "Give me a practice problem about torque."

In the past, you'd have to wait for a teacher to write one, or dig through a textbook. Now, the AI instantly spits out a problem. But here's the catch: What if the AI made up a problem that is impossible to solve, or one where the answer is wrong? That would be like a coach giving you a playbook with a hole in it.

This paper is about building a quality control inspector that lives inside the AI, checking its own work before it shows the problem to you.

The Big Problem: The "Hallucinating" Chef

Think of the AI as a very fast, very confident chef who can cook up a new recipe (a physics problem) in seconds.

• The Good: Sometimes, the chef makes a delicious, perfect dish.
• The Bad: Sometimes, the chef adds an ingredient that doesn't exist (like "a 5-ton fly") or writes a recipe that contradicts itself (like "boil the water at -10 degrees").

If you serve these bad dishes to students, they get confused and frustrated. The researchers wanted to know: Can we teach the AI to taste its own food and say, "Wait, this is burnt," before it gets to the student?

The Experiment: The Taste Test

The researchers set up a simulation with 34 physics students. They asked the AI to generate 543 practice problems.

1. The Expert Judge: A human physics professor (who has taught for 20+ years) looked at every single problem and graded it. He checked: Is this solvable? Is the answer right? Is the question clear? Is it too easy or too hard?
2. The Student Vote: The students were shown two AI-generated problems at a time and asked, "Which one do you want to try?" This told the researchers what students actually liked, even if they didn't know the answer yet.
3. The AI Judge: The researchers then asked different AI models to look at the same problems and try to grade them just like the human professor did.

The Findings: What Actually Matters?

The researchers were looking for a "magic checklist" of things the AI should check automatically. They found that you don't need a 100-item checklist. You just need a few key things.

Here are the three golden rules the AI needs to follow to make a good problem:

1. The "Roadmap" Check (Solution Strategy)

Analogy: Imagine asking for directions.

• Bad AI: "Go to the store." (No map, no turns, no idea where to start).
• Good AI: "Go to the store. First, turn left at the bank, then walk two blocks."
• The Finding: Students loved problems where the AI gave a tiny hint or a "roadmap" on how to start solving it (without giving away the answer). It made the problem feel less scary and more like a puzzle they could solve.

2. The "Clarity" Check (Specific & Complete)

Analogy: Ordering a pizza.

• Bad AI: "I want a pizza." (Do they want cheese? Pepperoni? Thin crust? How big?)
• Good AI: "I want a large pepperoni pizza with extra cheese."
• The Finding: The problem must have all the numbers and details needed to solve it. If the AI forgets to say "the car weighs 1000kg," the student is stuck. The AI must be specific.

3. The "Unit" Check (Clear Units)

Analogy: Buying fabric.

• Bad AI: "I need 5 of fabric." (5 what? Inches? Meters? Yards? You can't buy it without knowing).
• Good AI: "I need 5 meters of fabric."
• The Finding: In physics, numbers without units are meaningless. The AI must explicitly state if the answer should be in "seconds," "meters," or "Joules."

The "Secret Sauce": The AI as a Judge

The most exciting part of the paper is that the AI can do this checking itself.

• They tested three different AI models.
• They found that a specific, slightly cheaper AI model (called o3-mini/low) was surprisingly good at spotting errors and checking if the problem was clear.
• The Result: You don't need a human teacher to check every single problem. You can set up a "gatekeeper" AI that says, "This problem is broken, throw it away," or "This one looks good, show it to the student."

The Takeaway

The paper concludes that we don't need to overcomplicate things. To make AI-generated physics problems useful, we just need to ensure they are:

1. Solvable (No missing info).
2. Clear (Units and steps are defined).
3. Helpful (A little hint on how to start).

If the AI checks these three boxes, the problems are usually good enough for students to learn from. It's like a self-correcting homework machine that makes sure the "recipe" is safe to eat before serving it to the student.

In short: AI can be a great tutor, but only if we teach it to double-check its own homework first.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) can generate physics practice problems in real-time, offering a scalable alternative to instructor-created materials. However, the educational value of these AI-generated items is compromised by the lack of rapid, reliable post-generation vetting.

• The Core Issue: AI models often produce problems that are physically inconsistent, unsolvable, misaligned with course materials, or contain "hallucinations" (e.g., incorrect formulas or impossible physical scenarios).
• The Gap: Current validation methods rely heavily on human instructors, which is not scalable. Existing automated validation often uses synthetic data or lacks robust frameworks to filter low-quality outputs in real-time conversational settings.
• Objective: The study seeks to identify a minimal set of automated, reliable, and relevant metrics that can validate AI-generated physics problems before they are presented to students, ensuring technical soundness and user appeal without exhaustive human intervention.

2. Methodology

The study employed a mixed-methods approach combining human expert labeling, LLM-as-a-judge benchmarking, and machine learning analysis on real-world student data.

A. Data Collection & System

• Platform: The study utilized Ethel, a virtual teaching assistant ecosystem developed at ETH Zurich, enhanced with Retrieval-Augmented Generation (RAG) to access course materials.
• Participants: 34 introductory physics students preparing for exams.
• Interaction: Students engaged in a simulated exam-prep session, prompting the chatbot for practice problems. The system generated 543 problems in response to these prompts.
• Selection Mechanism: Students were presented with two candidate problems per prompt and forced to choose one to attempt. This "choice" served as the ground truth for relevance (student preference).

B. Ground Truth & Metrics

• Expert Labeling: One expert instructor (with 20+ years of experience) manually rated all 543 problems across a comprehensive set of ~25 quality metrics.
• Metric Categories:
  • Technical/Structural: Solvability, clarity, completeness, correct solution, standard notation.
  • Pedagogical: Bloom's taxonomy level, difficulty level, relevance to user request, inclusion of solution strategies.
  • Content Quality: Realistic data, bias-free language, grounding in textbook.
• Classification: Metrics were categorized as confirmatory (sufficient sample size per class, $n \ge 30$) or exploratory (skewed data).

C. Evaluation Framework

1. Reliability (RQ1): Three commodity LLMs (GPT-4o, GPT-4o-mini, GPT-o3-mini/low) acted as "judges" to predict the expert labels. Performance was measured using macro-averaged Precision, Recall, F1-score, and Accuracy.
2. Relevance (RQ2): Random Forest models were trained to predict the student's "chosen" problem based on the metric values. Feature importance scores identified which metrics drove student selection.
3. Automatic Assessability (RQ3): The study determined which metrics could be computed fast and cheaply enough for real-time deployment.

3. Key Contributions

• Empirical Dataset: Creation of a unique dataset of 543 authentic student-chatbot interactions with expert ground-truth labels, moving beyond synthetic data.
• Metric Validation: A systematic benchmarking of LLM-as-a-judge capabilities against human experts for physics problem generation.
• The "Compact Metric Stack": Identification of a minimal subset of metrics that balances reliability, relevance, and computational cost, providing a blueprint for real-time formative assessment.
• Operational Definitions: Clear definitions for "Reliable" (LLM-expert agreement), "Relevant" (predictive of student choice), and "Automatically Assessable" (feasible at scale).

4. Key Results

A. LLM Performance as Judges (Reliability)

• High Performance: Three metrics achieved strong agreement ($F1 \ge 0.80$) with human experts:
  1. includes-solution-strategy (F1: 0.838)
  2. measurement-unit-is-clearly-stated (F1: 0.835)
  3. llm-solution-is-correct (F1: 0.828)
• Moderate Performance: Structural clarity metrics like task-is-solvable and task-is-clear hovered around $F1 \approx 0.60$.
• Poor Performance: Cognitive depth metrics (bloom-level, exercise-difficulty-level) performed poorly ($F1 < 0.40$). The AI struggled to infer the intended difficulty from the prompt, often defaulting to "foundational" when humans could not determine the level.
• Model Comparison: The reasoning model GPT-o3-mini/low performed best overall (44% of top scores), though GPT-4o was competitive. Increasing reasoning depth did not always yield better results.

B. Predictors of Student Choice (Relevance)

• Random Forest Analysis: Students prioritized surface-level cues visible before solving:
  • Numerical Problems: The strongest predictors were measurement-unit-is-clearly-stated (0.364) and includes-solution-strategy (0.261).
  • Multiple-Choice Problems: bloom-level, exercise-difficulty-level, and distinct-misconceptions were top predictors.
  • Combined: includes-solution-strategy was the single most important feature across all problem types.
• Surprising Finding: llm-solution-is-correct was a significant predictor of choice, even though students cannot know the solution is correct before attempting the problem. The study attributes this to a correlation with surface clarity; problems that look clear and well-structured are more likely to have correct solutions.

C. The Optimal Metric Stack

The study converged on a four-metric stack that is both reliable (high LLM agreement) and relevant (predicts student choice):

1. includes-solution-strategy: Acts as a proxy for conceptual depth and engagement.
2. llm-solution-is-correct: Ensures technical validity.
3. task-is-specific-and-complete: Ensures the problem is solvable with provided info.
4. measurement-unit-is-clearly-stated: Ensures clarity for numerical tasks.

Note: Metrics related to difficulty and Bloom's level, while pedagogically important, were found to be unreliable for automated assessment in this context.

5. Significance and Implications

• Scalable Formative Assessment: The study demonstrates that exhaustive scoring is unnecessary. A "lightweight" validation layer using a small subset of metrics is sufficient to filter out defective problems and ensure user appeal.
• Cost-Efficiency: The proposed stack allows for real-time deployment. Structural checks can be performed by cheaper, low-latency models, while correctness checks can use reasoning models only when necessary.
• Pedagogical Insight: Students value clarity, solvability, and hints (solution strategies) over abstract difficulty ratings. They prefer problems that look "real" (units, clear prompts) and offer a roadmap for solving.
• Limitations & Future Work: The study is limited by a single-expert ground truth and a specific student cohort. Future work requires multimodal validation (for diagrams) and longitudinal studies to link these metrics to actual learning gains.

Conclusion: The paper provides a practical blueprint for deploying AI-generated practice in physics. By focusing on a curated core of structural and learner-visible checks, educational systems can ensure technical soundness and user engagement without the prohibitive cost of human review for every generated item.