Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning

Technical Summary: Response-Free Item-Difficulty Modelling for Multiple-Choice Items with Fine-Tuned Transformers

Problem Statement
Accurate estimation of item difficulty is critical for high-stakes educational assessment, yet conventional response-based calibration is often unavailable for newly written items, secure pre-tests, or small populations. While expert judgments offer a partial solution, they lack sufficient reliability. This necessitates response-free item-difficulty modelling (IDM), where difficulty is inferred directly from item wording prior to administration.

Existing approaches face significant limitations:

1. Feature Engineering Pipelines: These manually extract text features (e.g., readability, syntactic depth) and feed them into statistical models. This process requires extensive preprocessing (lemmatization, stopword removal) that discards morphosyntactic information and relies on theory-driven feature design.
2. Explanatory IRT with Frozen Encoders: Methods like the Linear Logistic Test Model (LLTM) integrate item features into the likelihood function. Recent adaptations use frozen BERT encoders to replace hand-coded features, but the representations remain tied to general-purpose pre-training objectives rather than the specific IDM task.
3. The Challenge of Reading Comprehension: For reading-comprehension multiple-choice questions (MCQs), difficulty arises from inferential demands across the passage, question, and options jointly. Simple feature extraction or frozen encoders often fail to capture these complex, cross-component interactions, especially in the small-sample regimes typical of applied psychometrics where labelled difficulty data is sparse.

Methodology
The authors propose and evaluate three transformer-based architectures that fine-tune encoder-only models (specifically ModernBERT) end-to-end on item wording, eliminating manual feature engineering. The study utilizes a nested training-size design on the RACE++ corpus (29,618 items), where difficulty targets are pseudo-labeled using a Large Language Model (Gemini 2.5 Flash) calibrated against human Rasch estimates from the PALRACE subset.

The three compared methods are:

1. Joint-Encoding (Baseline):

   • Architecture: The passage, question, and options are concatenated into a single input sequence separated by [SEP] tokens.
   • Mechanism: A standard Transformer encoder processes the flat sequence. The pooled representation from the [CLS] token is passed through a linear regression head to predict difficulty ($\hat{b}_i$).
   • Objective: Minimizes Mean Squared Error (MSE) between predicted and pseudo-labeled difficulty.

2. Component-Wise Encoding:

   • Architecture: The passage, question, and option set are encoded separately through a shared encoder.
   • Mechanism: Each component yields a [CLS] representation. These three vectors are concatenated and fed into the regression head.
   • Motivation: Explicitly enforces the functional decomposition of an MCQ, theoretically allowing the model to preserve relational features of distractors that might be lost in a flat sequence.

3. Multi-Task Learning (MTL):

   • Architecture: Retains the joint-encoding input but adds an auxiliary Multiple-Choice Question Answering (MCQA) branch.
   • Mechanism: The shared encoder feeds two heads:
     • A regression head for difficulty ($\hat{b}_i$).
     • A classification head ($f_\xi$) that predicts the probability of each option being correct.
   • Training: Tasks alternate per batch. A learnable task-conditioning vector ($z_t$) is added to the [CLS] embedding to signal the active task. The MCQA loss is weighted ($\lambda=0.05$) to prevent negative transfer.
   • Motivation: The auxiliary task acts as a regularizer, forcing the encoder to learn representations capable of option-level discrimination. This aligns the learned features with the cognitive process of test-taking, providing a supervisory signal when the primary difficulty label is sparse.

Key Results
The models were evaluated across three training-set sizes ($n \approx 800$, $n \approx 4,300$, and $n \approx 23,000$) using RMSE, $R^2$, and Spearman $\rho$.

• Viability of End-to-End Fine-Tuning: All transformer-based methods significantly outperformed the "dummy" regressor (which predicts the training-set mean difficulty). At $n \approx 800$, the models reduced RMSE by 14–16% compared to the baseline, demonstrating that transformer fine-tuning is a viable alternative to feature-engineering pipelines.
• Component-Wise Encoding: This variant showed no detectable benefit over joint encoding. In fact, it performed slightly worse on RMSE and $R^2$ at smaller sample sizes. The authors interpret this as evidence that the self-attention mechanism in joint encoding already effectively harvests cross-component signals, making explicit architectural decomposition redundant.
• Multi-Task Learning (MTL):
  • At the smallest sample size ($n \approx 800$), MTL delivered significant paired improvements over joint encoding across all metrics (RMSE, $R^2$, and Spearman $\rho$).
  • As the training set size increased ($n \approx 4,300$ and $23,000$), the advantage of MTL diminished and became statistically indistinguishable from joint encoding.
  • This confirms the hypothesis that the auxiliary MCQA objective provides a crucial regularizing signal specifically when the primary difficulty signal is sparse.

Significance and Claims
The paper claims that transformer fine-tuning, particularly when regularized by a suitable auxiliary task, recovers a substantial share of the wording-derivable difficulty signal at training-set sizes typical of applied measurement contexts.

Key takeaways include:

• Efficiency: End-to-end fine-tuning collapses the preprocessing and modelling stages of traditional feature engineering into a single, data-driven process.
• Small-Sample Regularization: In data-scarce regimes, adding an auxiliary task that mimics the test-taking process (MCQA) acts as an effective inductive bias, preventing the model from learning surface-level shortcuts and improving generalization.
• Practical Application: These response-free predictions can serve as priors or initial estimates in calibration workflows, potentially reducing the sample size required for subsequent response-based refinement.

The authors remain modest regarding the absolute ceiling of text-based prediction, acknowledging that factors like test-taking strategies and visual layout are not captured in the wording, and that the pseudo-labeled targets themselves contain noise from both the LLM and the underlying Rasch estimation. However, the framework provides a customizable interface for future psychometrically motivated extensions.