Synthetic Student Responses: LLM-Extracted Features for IRT Difficulty Parameter Estimation

Technical Summary: Synthetic Student Responses for IRT Difficulty Estimation

Problem Statement
Educational assessment development faces a significant bottleneck: determining the difficulty of new questions traditionally requires resource-intensive pre-testing with actual students followed by statistical analysis (Item Response Theory, or IRT). This process creates delays and costs that hinder teachers and large-scale assessment developers. While existing research attempts to predict difficulty directly from text using machine learning, most approaches treat it as a direct regression problem, bypassing the fundamental mechanism of how difficulty is established in practice: through patterns of student responses. Furthermore, current methods often rely on conventional linguistic features or embeddings, underutilizing the potential of Large Language Models (LLMs) to extract deep pedagogical insights.

Methodology
The authors propose a two-stage approach that simulates the student response process to derive difficulty parameters without new student testing.

1. Data and Feature Engineering:

   • Dataset: The study utilizes a dataset from Zapien, an adaptive math platform in Chile, containing 251,851 student responses to 4,696 unique math questions.
   • Benchmark: Ground truth difficulty parameters were established using a 1-Parameter Logistic (1PL) IRT model on the full dataset.
   • Feature Extraction: The model integrates five data sources:
     • Question Text Embeddings: Contextual vectors generated by ModernBERT, combining the problem statement and answer options.
     • Linguistic/Structural Features: Metrics such as word count, LaTeX presence, and option similarity.
     • LLM-Extracted Pedagogical Features: Using Google's Gemini models, the system extracts complex attributes including solution step counts, Bloom's Taxonomy levels, potential misconceptions, cognitive complexity, and distractor plausibility.
     • Student Embeddings: A learned 8-dimensional vector for each student to capture individual ability profiles.

2. Stage 1: Neural Network for Correctness Prediction:
   A neural network is trained to predict the probability of a specific student answering a specific question correctly. The architecture concatenates student embeddings, question text embeddings, and the engineered feature vectors (processed through dense layers with ReLU activation and dropout). The output is a probability score for each student-question pair.

3. Stage 2: IRT Parameter Estimation:
   The trained model is applied to a holdout set of 470 unseen questions. It generates a "synthetic" response matrix by predicting how the 1,875 students in the training data would answer these new questions. These probability outputs are binarized (thresholded at 0.5) to simulate correct/incorrect outcomes. A 1PL IRT model is then fitted to this synthetic matrix to estimate the difficulty parameters ($\beta$) for the unseen questions.

Key Results

• Prediction Accuracy: The neural network achieved an F1 score of 0.8350 and an AUC-ROC of 0.7789 in predicting student correctness on the holdout set.
• Difficulty Estimation: The derived difficulty parameters showed a Pearson correlation of 0.7791 and a Spearman rank correlation of 0.7606 with the benchmark parameters calculated from real student data.
• Feature Ablation:
  • Models using only text embeddings achieved a Pearson correlation of ~0.68.
  • Adding traditional linguistic and structural features provided negligible improvement.
  • The inclusion of LLM-extracted pedagogical features was the critical factor, boosting the correlation to ~0.78. This suggests that AI-derived insights into cognitive load and solution complexity are more predictive of difficulty than surface-level text metrics.
• Efficiency: An RMSE-based efficiency evaluation revealed that the model's accuracy (RMSE of 1.1857) is comparable to a traditional 1PL IRT analysis using approximately 5,818 real student responses (roughly 22.8% of the available data for those items).

Significance and Claims
The paper claims that it is possible to accurately estimate IRT difficulty parameters without conducting new student pre-tests by modeling the student response process. The primary contribution is the demonstration that combining semantic embeddings with LLM-extracted pedagogical features (such as solution steps and misconceptions) significantly outperforms traditional text-based approaches.

The authors argue that this simulation-based approach offers a more interpretable and robust alternative to direct regression, as it mirrors the actual educational process where difficulty emerges from student performance patterns. The study suggests this method could substantially reduce the resource burden and time required for developing high-quality assessments while maintaining psychometric validity.

Limitations
The authors acknowledge several constraints:

• Generalizability: The data is specific to the Chilean educational context, which may limit applicability to other regions.
• LLM Stability: The probabilistic nature of LLMs introduces potential variability in feature extraction, despite aggregation strategies.
• Interpretability: The neural network acts as a "black box," making it difficult to definitively isolate the contribution of specific features beyond the ablation study.
• Sensitivity: Results may vary based on the specific LLM vendor and inference parameters used.