💬 NLP

Difficulty-Controllable Multiple-Choice Question Generation Using Large Language Models and Direct Preference Optimization

This paper proposes a novel method for generating difficulty-controllable multiple-choice reading comprehension questions by leveraging a large language model trained with direct preference optimization to overcome the limitations of existing approaches in directly producing multiple-choice formats and accurately controlling question difficulty.

Original authors: Yuto Tomikawa, Masaki Uto

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuto Tomikawa, Masaki Uto

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to help 30 different students learn English. Some are beginners who need simple stories, while others are advanced learners ready for complex novels. In the real world, creating the perfect reading test for each student is a nightmare. It takes hours to write questions, and you have to guess if they are too easy or too hard.

This paper is about building a super-smart robot teacher that can instantly write the perfect reading test for any student, at exactly the right difficulty level.

Here is the story of how they built it, explained simply:

1. The Problem: The "One-Size-Fits-All" Robot

Before this study, robots (AI) could write reading questions, but they had two big flaws:

They couldn't make "Multiple Choice" questions: They mostly wrote questions where you had to find a specific sentence in the text. But in real schools, we almost always use multiple-choice questions (A, B, C, D). The old robots couldn't do this well.
They were bad at guessing difficulty: If you asked a robot, "Write an easy question," it might write something that was actually hard. It was like asking a chef to "make a spicy dish," and they accidentally made it taste like a fire extinguisher. They didn't have a way to guarantee the difficulty was right.

2. The Solution: The "Taste-Test" Training

The researchers used a powerful AI called Llama 3.1 (think of it as a very talented but untrained apprentice chef). To teach it how to control difficulty, they used a special training method called Direct Preference Optimization (DPO).

Here is the analogy:

Old Way (Supervised Learning): You show the apprentice a recipe book and say, "Copy this." They memorize the words but don't really understand why a dish is spicy or sweet.
New Way (DPO): You act like a strict food critic. You ask the apprentice to make two dishes:
1. A dish that is perfectly the right spice level.
2. A dish that is too spicy or too bland.
  Then you say, "I like the first one, I hate the second one."
  The apprentice learns by comparing the two. They don't just memorize; they learn the feeling of "perfect difficulty."

By doing this over and over, the robot learns to adjust the "spiciness" (difficulty) of the questions exactly to what you ask for.

3. The Secret Ingredient: The "Virtual Student" Army

How do you know if a question is actually easy or hard? You can't ask real students every time (that takes too long).

So, the researchers built an army of 77 "Virtual Students" (computer programs that are good at answering questions).

Some virtual students are geniuses (they get everything right).
Some are struggling (they only get 30% right).
Some are average.

When the robot writes a new question, it runs it through this army.

If the geniuses get it right but the struggling students get it wrong, it's a medium-difficulty question.
If even the geniuses get it wrong, it's super hard.
If even the struggling students get it right, it's super easy.

This data tells the robot exactly how "spicy" the question is, allowing it to fine-tune its training.

4. The Results: A Master Chef

The researchers tested their new robot against the old one.

Accuracy: When they asked for a "Level 2" question, the old robot gave them a "Level 5" or a "Level 0." The new robot (with the DPO training) gave them a question that was almost exactly "Level 2."
Quality: They worried that making the questions harder might make them sound weird or nonsensical. But the tests showed the new robot's questions were just as fluent and relevant as the old ones.
The "Few-Shot" Test: They tried to see if they could just tell the robot how to do it with a few examples (like showing it one easy question and saying "do this"). It failed. The robot couldn't learn the concept just by looking at examples; it needed the deep "taste-test" training (DPO) to truly understand difficulty.

The Bottom Line

This paper introduces a new way to teach AI to write school tests. Instead of just memorizing patterns, the AI learns to calibrate its output like a master chef adjusting a recipe.

Now, instead of a teacher spending all night writing tests, they can simply tell the AI: "Write a multiple-choice question about this story that is perfect for a student who is 60% ready for the next grade." And the AI will do it instantly, accurately, and without breaking a sweat.

1. Problem Statement

The paper addresses two critical limitations in existing automated question generation (QG) systems for reading comprehension:

Inability to Generate Multiple-Choice Questions (MCQs): Conventional difficulty-controllable QG methods primarily generate extractive questions (where the answer is a text span from the passage). They cannot directly generate MCQs, which are the standard format in educational assessments.
Suboptimal Difficulty Control Accuracy: Existing methods typically train models using Supervised Fine-Tuning (SFT) to maximize the conditional likelihood of generating a question given a difficulty label. However, they do not explicitly optimize for the accuracy of the difficulty control itself. Consequently, the generated questions often deviate from the intended difficulty levels, limiting their utility in adaptive learning systems.

2. Methodology

The authors propose a novel framework that combines Large Language Models (LLMs) with Direct Preference Optimization (DPO) to generate MCQs with precise difficulty control.

A. Dataset Construction

Since standard datasets (like RACE) lack explicit difficulty labels, the authors constructed a specialized dataset:

IRT-Based Difficulty Estimation: They utilized Item Response Theory (IRT), specifically the Rasch model, to estimate difficulty levels ( $b$ ).
Virtual Learners: Instead of using human data (which is costly), they constructed 77 diverse "virtual learners" using various pre-trained transformer models (e.g., BERT, RoBERTa, DeBERTa) with varying amounts of training data to simulate different ability levels ( $\theta$ ).
Process: These QA systems answered existing questions, and their response data was used to estimate the difficulty parameter ( $b$ ) for each question, creating a dataset of quadruples: $(\text{Context}, \text{Question}, \text{Options}, \text{Difficulty})$ .

B. Model Architecture

The base model is Llama 3.1 (8B), trained in two stages:

Supervised Fine-Tuning (SFT): The model is initially fine-tuned on the constructed dataset to learn the format of generating an MCQ (question + 4 options) given a context and a target difficulty level.
- Input: Context + Target Difficulty ( $b$ ).
- Output: Question text + Correct Option + 3 Distractors.
Direct Preference Optimization (DPO): To explicitly optimize difficulty control, the authors apply DPO, a technique derived from Reinforcement Learning from Human Feedback (RLHF) but more stable and efficient.
- Objective: Maximize the reward function defined as the negative squared error between the specified difficulty and the estimated difficulty of the generated question.
- Preference Pairs: For a given input (context + target difficulty), the "preferred" output ( $y_w$ ) is the question generated by the model that matches the target difficulty best. The "dispreferred" output ( $y_l$ ) is a question generated for the same context but with a mismatched difficulty (or randomly selected from the dataset).
- Loss Function: The model is trained to increase the likelihood of $y_w$ relative to $y_l$ , effectively aligning the model's output distribution with the desired difficulty control objective without needing explicit reward modeling or PPO.

3. Key Contributions

First MCQ-Specific Difficulty Control: The study is the first to propose a method that directly generates multiple-choice questions with controllable difficulty levels, moving beyond extractive answers.
DPO for Difficulty Alignment: The authors introduce DPO as a mechanism to explicitly optimize the accuracy of difficulty control, addressing the gap left by standard SFT approaches.
IRT-Integrated Framework: They provide a robust pipeline for constructing training datasets with IRT-based difficulty labels using virtual learners, making the approach scalable without massive human annotation.

4. Experimental Results

The method was evaluated on the RACE dataset (87,866 instances). The proposed model ( $QG_{dpo}$ ) was compared against the SFT-only baseline ( $QG_{sft}$ ).

Difficulty Control Accuracy:
- Correct Response Rate: $QG_{dpo}$ showed a much stronger correlation between the specified difficulty and the actual correct response rate of the virtual learners compared to $QG_{sft}$ .
- IRT Estimation Error: The Mean Absolute Error (MAE) between specified and estimated difficulty was significantly lower for $QG_{dpo}$ (1.07) compared to $QG_{sft}$ (1.41).
- Fisher Information: Questions generated by $QG_{dpo}$ yielded higher Fisher Information near the target ability level, indicating they are more effective for precise ability estimation in adaptive testing.
Question Quality:
- Using an LLM-as-a-Judge approach (GPT-4.1 mini), the authors evaluated fluency, content relevance, and answerability.
- Result: $QG_{dpo}$ maintained high scores in fluency and relevance, comparable to the SFT baseline and human-created questions. While answerability remained a challenge for both (lower than human questions), DPO did not degrade the quality.
Comparison with Few-Shot Learning:
- The authors tested if providing examples (few-shot) to a base LLM (GPT-4.1 mini) could achieve difficulty control.
- Result: Few-shot learning failed to establish a correlation between specified and generated difficulty, confirming that fine-tuning with DPO is necessary for this task.
Qualitative Analysis:
- As difficulty increased, the proportion of complex reasoning types (e.g., multi-sentence reasoning) increased in $QG_{dpo}$ more distinctly than in $QG_{sft}$ , suggesting the model learned to adjust reasoning complexity to match difficulty targets.

5. Significance and Future Work

Educational Impact: This method enables the creation of adaptive learning systems where questions can be dynamically generated to match a learner's specific ability level, optimizing the learning curve (targeting ~50% success rate).
Scalability: The use of IRT and virtual learners allows for the creation of difficulty-labeled datasets without expensive human testing.
Limitations & Future Directions:
- Answerability: The generated questions still occasionally lack a single correct answer. The authors suggest using external LLMs to provide feedback on answerability as a reward signal for future DPO iterations.
- Data Scarcity: The current method relies on large datasets (RACE). Future work will explore data augmentation for low-resource educational settings.
- Adaptive Testing Validation: Future studies will evaluate the method in real-world adaptive testing scenarios to measure actual improvements in learner ability estimation efficiency.

In conclusion, the paper demonstrates that integrating Direct Preference Optimization with Large Language Models significantly enhances the controllability of question difficulty in MCQ generation, offering a viable technical solution for next-generation adaptive educational technologies.