💬 NLP

Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models

This paper introduces ReQUESTA, a hybrid multi-agent framework that orchestrates large language models with rule-based components to systematically generate cognitively diverse and psychometrically superior multiple-choice questions, outperforming single-pass zero-shot baselines in difficulty, discrimination, and alignment with reading comprehension goals.

Original authors: Yu Tian, Linh Huynh, Katerina Christhilf, Shubham Chakraborty, Micah Watanabe, Tracy Arner, Danielle McNamara

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Tian, Linh Huynh, Katerina Christhilf, Shubham Chakraborty, Micah Watanabe, Tracy Arner, Danielle McNamara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Shot" Chef

Imagine you ask a very talented, super-smart chef (a Large Language Model, or LLM) to cook a complex three-course meal. You give them a single instruction: "Make me a meal with an appetizer, a main course, and a dessert."

The chef might make a delicious meal, but they might also:

Forget to check if the dessert is too sweet.
Serve a main course that is slightly undercooked.
Make the appetizer look nothing like the main course.

In the world of education, this "one-shot" chef is how computers usually create Multiple-Choice Questions (MCQs). They are good at making simple questions (like "What color is the sky?"), but they struggle to make tricky, high-quality questions that test deep thinking (like "Why did the character make that choice?"). They often produce questions that are too easy, have confusing answers, or have "distractors" (the wrong answers) that are obviously wrong.

The Solution: ReQUESTA (The Restaurant Kitchen)

The authors of this paper built a system called ReQUESTA. Instead of asking one chef to do everything at once, they turned the kitchen into a highly organized restaurant with a team of specialized staff.

Think of ReQUESTA not as a single robot, but as a conductor leading an orchestra, or a movie director managing a film crew.

Here is how their "kitchen" works:

The Preprocessor (The Prep Cook): Before anyone starts cooking, this agent chops the text into manageable chunks. It makes sure the ingredients are ready.
The Planner (The Head Chef): This agent reads the text and writes a detailed recipe. It decides: "We need a question about the facts here, a question about what the character is thinking, and a question about the main theme." It doesn't cook yet; it just plans.
The Generators (The Line Chefs): There are three different chefs, each with a specific job:
- Chef A only makes "Fact" questions (What happened?).
- Chef B only makes "Inference" questions (Why did it happen?).
- Chef C only makes "Big Idea" questions (What is the main point?).
- Why separate them? Because asking one chef to do all three at once often leads to mistakes. Specialization makes them better at their specific task.
The Evaluator (The Food Critic): Once a dish is made, it doesn't go straight to the customer. The Critic tastes it. Is the question clear? Are the wrong answers (distractors) tricky enough to fool someone who didn't study, but not so tricky that they fool everyone? If the dish is bad, the Critic sends it back to the Line Chef with notes on how to fix it.
The Formatter (The Plating Specialist): Finally, this agent makes sure all the plates look the same size and the letters (A, B, C, D) are aligned perfectly.

The Experiment: The Taste Test

To see if this "team kitchen" was better than the "one-shot chef," the researchers set up a massive taste test.

The Setup: They took 20 academic articles (like textbook chapters) and asked two systems to create questions for them.
- System A (ReQUESTA): The team kitchen with the planner, specialized chefs, and critic.
- System B (GPT-5 Baseline): The "one-shot" chef who just got a single prompt to "make questions."
The Tasters: 572 real people read the articles and answered the questions.
The Judges: Expert human raters also looked at the questions to grade them on quality.

The Results: The Team Kitchen Wins

The results showed that the ReQUESTA team kitchen produced much better food (questions) than the single chef.

Harder and Smarter Questions: The questions made by ReQUESTA were harder to answer correctly. This isn't a bad thing! It means the questions were actually testing if the student understood the material, rather than just guessing. They were better at telling the difference between a student who knew the material and one who didn't.
Better "Wrong Answers" (Distractors): In a multiple-choice question, the wrong answers need to be believable. If the wrong answers are silly, anyone can guess the right one.
- The "one-shot" chef often made silly wrong answers.
- ReQUESTA's "Critic" and "Specialized Chefs" made wrong answers that sounded very real. They were linguistically consistent (they looked and sounded like the right answer) and semantically plausible (they made sense in the context).
Sticking to the Main Idea: ReQUESTA's questions were more focused on the most important parts of the text. The single chef often got distracted by tiny, unimportant details.

The Big Takeaway

The most important lesson from this paper isn't about having a "smarter" computer brain. The researchers used the same powerful AI model (GPT-5) for both systems.

The difference was how they organized the work.

Old Way: "Do everything at once, quickly." (Single-pass prompting)
New Way (ReQUESTA): "Plan first, assign specialists, check the work, fix mistakes, and then present." (Agentic orchestration)

The paper proves that you don't necessarily need a bigger, more expensive AI to get better results. You just need a better workflow. By breaking a big, messy job into small, controlled steps and having different "agents" check each other's work, you can create high-quality, reliable educational tools that a single AI prompt just can't match.

Technical Summary: ReQUESTA – A Hybrid Multi-Agent Framework for Cognitively Diverse MCQ Generation

Problem Statement
While Large Language Models (LLMs) have made automated Multiple-Choice Question (MCQ) generation increasingly feasible, reliably producing items that satisfy controlled cognitive demands remains a significant challenge. Current single-pass prompting approaches often struggle to align generated questions with higher-order cognitive skills (e.g., inference, synthesis) and frequently produce items with inconsistent distractor quality, lack of structural parallelism, or superficial alignment with source texts. Existing hybrid approaches have yet to demonstrate reliable control over the cognitive complexity of generated questions without additional structural constraints. The core problem addressed is how to move beyond monolithic, single-pass generation to create a system that can systematically target specific cognitive levels (text-based recall, inferential reasoning, and main idea synthesis) while ensuring high psychometric and pedagogical quality.

Methodology: The ReQUESTA Framework
The authors introduce ReQUESTA (Reading Question-generation Using Educational Smart Text Agents), a hybrid, multi-agent framework designed to orchestrate MCQ generation through a coordinated sequence of LLM-powered agents and rule-based components. Rather than relying on model scaling alone, ReQUESTA decomposes the authoring process into specialized subtasks to ensure controllability and reliability.

Architecture and Workflow:
- Preprocessor (Rule-based): Segments input text into coherent, manageable units using sentence-level parsing.
- Planner (LLM-powered): Analyzes segments to identify key concepts, implicit inferences, and central themes. It generates a structured plan (intermediate representation) specifying which segments and elements will be used for different question types (text-based, inferential, main idea).
- Controller (Rule-based): Interprets the Planner's output and orchestrates task execution, assigning specific text segments and cognitive targets to the appropriate generator agents.
- Question Generators (LLM-powered): Three specialized agents generate questions based on the Planner's constraints:
  1. Text-based: Targets explicit facts.
  2. Inferential: Targets relationships requiring integration of information.
  3. Main Idea: Targets overarching themes.
- Evaluator (LLM-powered): Assesses generated items against a checklist (clarity, stem-answer alignment, distractor plausibility). Items failing criteria are returned to the generator for iterative revision.
- Formatter (Rule-based): Standardizes output, shuffling options and enforcing consistent labeling.
- Option-Shortening Module: A post-processing step using a collaborative agent system (Syntax Analyzer, Length Determiner, Candidate Generator, Candidate Selector) to correct length imbalances among distractors, a common failure mode in LLM generation.
Prompt Engineering Strategies:
The framework employs four key strategies to guide agent behavior:
1. Persona-based Prompting: Assigns specific roles (e.g., "experienced college instructor") to align outputs with assessment norms.
2. Chain-of-Thought (CoT): Requires agents to report intermediate reasoning steps, making decision processes explicit and inspectable.
3. Few-shot Prompting: Provides examples of effective vs. ineffective outputs (e.g., for option shortening) to calibrate stylistic constraints.
4. Self-critique: Agents are prompted to review their own outputs for common weaknesses (ambiguity, misalignment) before external evaluation.
Experimental Design:
The framework was evaluated against a GPT-5 zero-shot baseline (single-pass prompting) using 20 academic expository passages from OpenStax textbooks.
- Participants: 572 adults recruited via Prolific.
- Task: Participants read passages and answered 5 MCQs per passage (2 text-based, 2 inferential, 1 main idea).
- Conditions: Within-participant design where participants answered items generated by either ReQUESTA or the GPT-5 baseline.
- Metrics:
  - Psychometric: Item difficulty ( $p$ ), discrimination ( $D$ ), and point-biserial correlation ( $r_{pb}$ ) analyzed via Classical Test Theory and mixed-effects logistic regression.
  - Expert Evaluation: Two expert raters scored 200 items (100 per method) on a rubric covering answer correctness, distractor incorrectness, topic relevance, writing clarity, and distractor quality (linguistic features, semantic plausibility, uniqueness).

Key Results

Psychometric Performance: ReQUESTA-generated items were consistently more challenging and discriminative than the GPT-5 baseline.
- Difficulty: ReQUESTA items had a lower mean difficulty index ( $M = 0.786$ ) compared to GPT-5 ( $M = 0.902$ ), indicating they were harder for learners.
- Discrimination: ReQUESTA items showed significantly higher discrimination ( $M = 0.39$ vs. $0.17$), better differentiating between high- and low-performing readers.
- Alignment: ReQUESTA items showed stronger point-biserial correlations with total test scores ( $M = 0.323$ vs. $0.275$), indicating better alignment with overall reading comprehension ability.
- Regression Analysis: Mixed-effects models confirmed that question source (ReQUESTA) was a significant predictor of lower accuracy ( $\beta = -1.277, p < .001$ ), holding question type and vocabulary constant.
Expert Evaluation:
- Topic Relevance: ReQUESTA items were rated significantly higher for alignment with central concepts ( $p < .05$ ).
- Distractor Quality: ReQUESTA outperformed the baseline in distractor linguistic features (consistency in length/structure) and distractor semantic plausibility (reasonableness to learners who missed the point), particularly for inferential questions.
- Writing Clarity: The GPT-5 baseline received higher scores for writing clarity ( $p < .001$ ), suggesting a trade-off where ReQUESTA's focus on cognitive rigor and complex distractors sometimes resulted in slightly less concise stems or options.
- Correctness: Both methods achieved near-perfect scores for answer correctness and distractor incorrectness.

Significance and Contributions
The paper claims three primary contributions to the field of LLM-based generation systems:

Workflow Design over Model Scaling: The study demonstrates that meaningful improvements in automated MCQ generation can be achieved through workflow and orchestration design rather than by scaling model parameters. By holding the underlying LLM (GPT-5) constant and introducing structured coordination across planning, generation, evaluation, and revision, ReQUESTA achieved superior outcomes compared to a single-pass baseline.
Controllable Generation Framework: ReQUESTA proposes a hybrid architecture that combines the expressive power of LLMs with the reliability of deterministic, rule-based processing. By explicitly engineering agent boundaries, task specialization, and iterative feedback loops, the system regulates the reasoning demands and structural constraints of generated artifacts, moving beyond the limitations of monolithic prompting.
Evaluation Practice: The study advances evaluation practices for structured generation systems by utilizing functional outcome measures (e.g., item difficulty, discrimination, distractor plausibility) rather than relying solely on surface-level indicators like grammaticality.

Conclusion
The authors conclude that agentic orchestration can systematically improve the reliability and controllability of LLM-based generation. ReQUESTA illustrates that decomposing complex tasks into auditable intermediate steps and enforcing constraints through repeated checks yields more reliable, cognitively diverse, and pedagogically aligned artifacts than single-pass prompting. The framework positions itself as a mature, evaluation-driven approach for automated assessment generation, highlighting that the design of the generation workflow is a key lever for structured artifact generation.