Scalable Generation and Validation of Isomorphic Physics Problems with GenAI

Technical Summary: Scalable Generation and Validation of Isomorphic Physics Problems with GenAI

Problem Statement
Traditional synchronous, single-attempt physics exams face increasing challenges regarding academic dishonesty, accessibility for non-traditional students, and the inability to share assessment items openly for practice. While asynchronous, multi-attempt assessments using large banks of isomorphic problems (items assessing the same underlying concepts and solution structures but varying in surface features) offer a promising alternative, creating and validating these banks at scale remains resource-intensive. Manual generation of high-quality variants is prohibitively expensive, and ensuring psychometric uniformity (consistent difficulty) across variants is difficult. Traditional validation relies on student response data, which is often sparse for individual items in randomized exams and only available post-deployment, leaving a "cold-start" problem for pre-deployment quality assurance.

Methodology
The authors propose a human-in-the-loop Generative AI (GenAI) framework for the scalable creation and validation of isomorphic physics problems, specifically within introductory Newtonian mechanics.

1. Generation Framework:

   • Prompt Chaining & Tool Use: The system decomposes problem generation into sequential sub-tasks using prompt chaining. It distinguishes between structural variations (construct-relevant changes like numerical values, spatial arrangements, and object counts) and contextual variations (surface features like scenarios and wording).
   • Control Mechanisms: Structural variations are constrained using Python code execution (via a tool interpreter) to enforce physical laws and mathematical bounds (e.g., ensuring forces balance or values remain positive). Contextual variations are generated with greater creative flexibility but guided by constraints on reading level and cultural relevance.
   • Workflow: The process involves identifying a template, defining components, generating variations via a seven-step prompt chain, and verifying correctness using LLMs.

2. Dataset Construction (ESTELA-Physics):

   • The framework produced ESTELA-Physics, a dataset of 29 isomorphic problem banks covering 12 topics in University Physics I (e.g., kinematics, forces, energy, momentum).
   • The dataset contains 666 individual items, formatted for import into learning management systems (Canvas), including numerical response, multiple-choice, and categorization formats.

3. Validation Strategy:

   • Post-Deployment (Student Data): 20 banks were deployed in three midterm exams for a class of >220 students. Difficulty homogeneity was assessed using item-level accuracy, standard deviation, and the extended Fisher's exact test.
   • Pre-Deployment (LM Simulation): 17 open-source Language Models (LMs) ranging from 0.6B to 32B parameters were used to simulate student responses on the same banks. LM accuracy was defined as the proportion of models answering correctly.
   • Correlation & Diagnosis: The study analyzed the correlation between LM and student performance and conducted case studies on heterogeneous banks to diagnose specific causes of difficulty (e.g., ambiguous wording, overlooked constraints).

Key Contributions

1. GenAI-Based Problem Generation Framework: A reproducible, human-in-the-loop workflow utilizing prompt chaining and tool use to generate isomorphic physics problems with explicit control over structural integrity and diverse contextual variations.
2. ESTELA-Physics Resource: A publicly available dataset of 29 isomorphic problem banks (666 items) across 12 physics topics, serving as a benchmark for scalable isomorphic generation and validation.
3. Empirical Validation Study: A dual-evaluation approach demonstrating that:
   • GenAI can generate banks with statistically homogeneous difficulty (75% of banks passed homogeneity tests using student data).
   • LM simulation can serve as a low-cost pre-deployment screening tool, identifying 70% of banks as homogeneous and flagging potential difficulty outliers.
   • LM responses can diagnose specific causes of unintended difficulty, such as implicit world knowledge gaps, overlooked verbal constraints, and ambiguous problem text.

Results

• Difficulty Homogeneity: Among 20 deployed banks, 75% were classified as homogeneous based on student response data (p > 0.05). LM simulation classified 70% of evaluated banks as homogeneous.
• LM-Student Alignment: Correlation between LM and student performance varied. Five of 13 overlapping banks showed moderate positive alignment (Pearson > 0.30). Alignment was stronger for banks with moderate difficulty and numerical formats but weaker for image-based questions or near-ceiling performance scenarios.
• Diagnostic Capability: In case studies of heterogeneous banks (6-1 and 9-1), LM errors successfully identified genuine difficulty outliers.
  • Bank 6-1: LMs revealed failures to apply implicit real-world knowledge (e.g., drag forces acting during net stretching), overlooked verbal conditions (e.g., distance traveled within a mud patch), and ambiguous problem text (e.g., undefined reference points for height).
  • Bank 9-1: LMs identified a specific geometric configuration (a ring pivoted at its edge) that consistently triggered errors in applying the parallel-axis theorem, mirroring student struggles.

Significance
The paper claims that GenAI can make the large-scale generation of isomorphic physics problems practical, enabling assessment formats where problem banks are openly available for practice without compromising diagnostic validity. Furthermore, the study positions LM-based simulation not as a perfect replacement for human review, but as a valuable, low-cost pre-deployment screening tool. It can flag potentially problematic variants and diagnose the root causes of difficulty heterogeneity (such as wording ambiguities or missed constraints) before items are administered to students, thereby complementing traditional human review and reducing the resource burden of creating robust assessment banks.