Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations

1. Problem Statement

Standardized educational assessments (e.g., NAEP) rely on expensive, time-consuming human pilot studies to determine the difficulty of test items. Traditional methods involve expert cognitive analysis or retrospective psychometric studies on real student data.

• Limitations of Current Approaches:
  • Cost & Scalability: Human pilots cost thousands of dollars and take weeks.
  • Failure of Direct Estimation: Automated text-based difficulty estimation (QDET) and direct prompting of LLMs to predict difficulty scores yield weak correlations with real-world student performance.
  • The Core Issue: Difficulty is not an intrinsic property of the text but depends on who answers the question. To accurately estimate difficulty using Item Response Theory (IRT), individual-level response data (correct/incorrect) is required, not just aggregate difficulty labels.
• Goal: To investigate whether open-source Large Language Models (LLMs) can simulate realistic student responses to generate psychometrically valid difficulty estimates without requiring costly human pilot studies.

2. Methodology

The authors propose a Simulation-Based Approach where LLMs act as "silicon students" to generate individual response data, which is then analyzed using IRT.

A. Data Source

• Dataset: 631 multiple-choice math questions from the National Assessment of Educational Progress (NAEP) for grades 4, 8, and 12.
• Ground Truth: Real-world student performance statistics (percentage correct) and demographic breakdowns provided by the NAEP.

B. Simulation Framework

The system simulates a "classroom" of $N$ students with varying proficiency levels.

1. Role-Playing Prompts: LLMs are prompted to adopt specific personas defined by:
   • Grade Level: 4, 8, or 12.
   • Skill Level: Mapped to NAEP descriptors (Below Basic, Basic, Proficient, Advanced).
   • Identity Attributes: To test the impact of persona grounding, the study varied identifiers:
     • No Identifier.
     • Student IDs (alphanumeric).
     • Single Name (all students share one name).
     • Diverse Names: Names stratified by gender and race/ethnicity (Asian, Black, Hispanic, White).
2. Classroom Composition: Simulated classrooms follow a non-uniform skill distribution reflecting NAEP trends: 25% Below Basic, 35% Basic, 25% Proficient, 15% Advanced.
3. Response Generation: The LLM generates a binary response (Correct/Incorrect) for each simulated student on each question.
4. IRT Modeling: The binary responses are fed into a Rasch model (1-Parameter IRT) to estimate:
   • Student Ability ($\beta$): Group-level ability for each skill tier.
   • Item Difficulty ($\delta$): The estimated difficulty of each question.

C. Models Evaluated

The study tested 10 open-source LLMs across different families and sizes:

• Gemma: 2-9B, 2-27B, 3-4B, 3-12B, 3-27B.
• Llama: 3.1-8B, 3.3-70B.
• Qwen: 2.5-14B, 2.5-32B, 3-4B.
• Baselines: Word2Vec, BERT, DistilBERT (for text-based difficulty), and Direct Prompting (asking the LLM to predict % correct directly).

D. Evaluation Metrics

• Correlation: Pearson ($r$) and Spearman ($\rho$) correlations between simulated success rates and real-world NAEP success rates.
• Discrimination: Area Under the Curve (AUC) to measure how well the IRT-estimated difficulty discriminates between easy and hard items.
• Distractor Analysis: Checking if simulated students select the same incorrect answers (distractors) as real students.

3. Key Contributions & Findings

A. Simulation Outperforms Direct Estimation

• Direct Prompting Failure: Asking LLMs directly to estimate the percentage of students who would answer correctly resulted in near-zero correlations ($r \approx 0$) across all models and grades.
• Simulation Success: Simulating individual student responses and aggregating them yielded strong correlations.
  • Grade 4: $r = 0.75$
  • Grade 8: $r = 0.76$
  • Grade 12: $r = 0.82$
• IRT Validity: The IRT-estimated difficulty parameters showed high AUC scores (0.77–0.90), proving the simulations capture the psychometric structure of the items, not just surface-level text complexity.

B. The "Weaker Model" Paradox

• Counter-Intuitive Finding: Models with lower mathematical reasoning capabilities (e.g., Gemma-2-9B, Gemma-3-27B) produced better difficulty simulations than highly proficient models (e.g., Llama-3.3-70B, Qwen2.5-32B).
• Reasoning: Highly capable models struggle to authentically simulate the error patterns of struggling students; they tend to "solve" the problem correctly even when prompted to be "Below Basic." Models that find the math challenging themselves are better at reproducing the distribution of student errors.

C. Impact of Identity and Diversity

• Diverse Names Improve Accuracy: Using diverse names (stratified by race and gender) consistently outperformed baselines with no identifiers or single homogeneous names.
• Mechanism: While the exact mechanism is under investigation, the authors suggest that diverse names provide better prompt grounding, reducing over-generalization and leading to more stable IRT parameter estimates.
• Note: The study emphasizes these are aggregate statistical observations, not predictions about specific individuals.

D. Content Area and Class Size

• Content Variance: Simulations performed best on Measurement problems ($\rho \approx 0.75-0.88$) but struggled with Algebra ($\rho \approx 0.45-0.64$), likely due to the complexity of algebraic reasoning errors.
• Class Size: Increasing the simulated classroom size from 50 to 300 students improved correlation strength, though diminishing returns were observed after 100 students.

E. Limitations on Distractors

• While the simulations accurately predict whether a student gets a question right or wrong, they are poor at predicting which specific wrong answer (distractor) a student will choose. Match rates for top distractors were only 31–47% (slightly above chance), indicating a gap in modeling specific misconception patterns.

4. Significance and Implications

• Cost-Effective Screening: This approach offers a low-cost, scalable alternative to human pilot studies for preliminary item screening. It allows educators to identify difficult or easy items before investing in expensive human testing.
• Psychometric Validity: The study demonstrates that LLMs can generate data that satisfies the requirements of Item Response Theory, bridging the gap between generative AI and psychometrics.
• Model Selection Strategy: The findings challenge the assumption that "bigger is better" for simulation tasks. For simulating human struggle, models with moderate capabilities may be more effective than state-of-the-art reasoning models.
• Ethical Considerations: The paper addresses the risks of using demographic names, acknowledging potential stereotype activation but framing the use of diverse names as a controlled variable to improve aggregate simulation fidelity rather than to make individual predictions.

Conclusion

The paper establishes that LLM-based student simulation is a viable, high-fidelity method for estimating math question difficulty. By simulating a diverse classroom of students and applying IRT, researchers can achieve correlations with real-world data as high as 0.82. The most effective strategy involves using open-source models with moderate math abilities, diverse identity prompts, and simulated classroom sizes of at least 100 students.