🤖 AI

Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

This pilot study demonstrates that while orchestrating multiple LLM agents can efficiently generate high-quality surface-level SAT Math multiple-choice questions, the resulting artifacts still lack the depth and cognitive rigor of expert-vetted benchmarks, thereby shifting the human researcher's role from direct authoring to specification, orchestration, and governance.

Original authors: Yuan An

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuan An

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years perfecting a specific recipe: creating the perfect multiple-choice questions for a high-stakes math exam (like the SAT). Traditionally, to make 1,000 of these questions, you would have to spend months writing them by hand, checking every fact, and grading them yourself. It's a slow, exhausting process.

This paper is about a pilot study where a researcher tried something radical: What if I don't write the questions myself? What if I become the "Head Chef" who manages a team of AI robots to do the cooking?

Here is the story of that experiment, broken down into simple concepts.

1. The Setup: The AI Kitchen

The researcher didn't just ask an AI, "Write me a math question." Instead, they built a complex, automated kitchen (a "workflow") with specific roles:

The Librarian (AI Agent 1): Scanned thousands of pages of open-source math textbooks and turned them into digital chunks.
The Architect (AI Agent 2): Took a real SAT question and found the exact page in the textbook where the answer lives.
The Cooks (AI Agents 3 & 4): Two different AI models (Gemini and GPT) were given the textbook pages and told, "Make a new question that looks like this one, but is slightly different."
The Food Critics (AI Judges): Two other AIs tasted every single question (both the original human ones and the new AI ones) and rated them on a 24-point checklist.

2. The Results: The "Taste Test"

The researchers generated over 1,000 new questions and compared them to the official SAT questions.

The Good News (Surface Level):
The AI cooks were amazing at the basics. The questions were grammatically perfect, had no typos, and the answers were clearly stated. If you just glanced at them, they looked just as good as the human-made ones. In fact, the AI judges gave the AI-generated questions higher scores for clarity than the human questions!

The Bad News (The "Secret Sauce"):
When the critics looked deeper, they found the AI was missing the "soul" of a good question.

Depth: The AI questions were often too shallow. They tested if you could memorize a fact, but not if you could think through a complex problem.
Difficulty: The AI struggled to hit the "Goldilocks" zone. Some were too easy, some too hard, and they didn't match the specific difficulty level they were supposed to have.
Tricky Options: In multiple-choice questions, the wrong answers (distractors) need to be plausible but clearly wrong. The AI often made the wrong answers too obvious or too confusing.

The Verdict: The AI can write a decent question, but it cannot yet write a perfect question that challenges a student's brain the way a human expert can. They are "good enough" for practice, but not quite ready to replace the experts.

3. The Real Discovery: The Job Change

The most interesting part of the paper isn't about the math questions; it's about what the human researcher actually did during the 10 days of the experiment.

The Old Way (The Solo Chef):

Time: 6 months.
Work: Writing every sentence, checking every fact, formatting every page.
Role: The "Doer."

The New Way (The AI Orchestrator):

Time: 10 days.
Work: The researcher spent 52% of their time not writing, but fixing the kitchen. They had to:
- Tell the robots exactly what to do (Specification).
- Fix it when a robot got stuck or hallucinated (Error Control).
- Design the checklist the critics used (Rubric Design).
- Decide if the final results made sense (Governance).
Role: The "Manager" or "Conductor."

The Big Metaphor: From Carpenter to Architect

Think of it like building a house.

Before AI: You were a carpenter. You spent months hammering nails, sawing wood, and laying bricks. You did the physical work.
With AI: You are now the Architect and Site Manager. You don't touch the wood. Instead, you spend your time drawing the blueprints, ordering the materials, making sure the robots are building the walls straight, and inspecting the final product to make sure the roof doesn't leak.

The paper argues that this is the future of scientific work. We aren't being replaced; our jobs are upgrading. We are moving from "content creators" to "quality controllers."

Why This Matters

Speed: A project that used to take a PhD student six months can now be done in 10 days. This means science can move much faster.
New Skills: To do this, researchers need to learn new skills. They need to know how to talk to AI, how to spot when AI is lying (hallucinating), and how to build systems that catch mistakes. This new job title might be called "AI Research Operations."
The Human Touch: Even though AI is fast, it still needs a human to be the "conscience" of the project. The AI can generate the questions, but only a human can decide if those questions actually teach the right things.

In short: AI is a powerful engine, but it still needs a skilled driver to steer it, check the oil, and decide where to go. The paper shows that while the engine is getting faster, the driver's job is becoming more important, not less.

1. Problem Statement

The paper addresses two interconnected challenges:

The Shift in Scientific Labor: While Large Language Models (LLMs) are increasingly capable of generating content, there is limited empirical evidence on how they fundamentally reshape the nature of scientific work. The authors posit that the researcher's role is shifting from "authoring content" to "defining, orchestrating, and verifying" AI-driven pipelines.
The Quality of AI-Generated Educational Content: In the context of standardized testing (specifically SAT Math), it remains unclear whether LLMs can generate Multiple Choice Questions (MCQs) that are pedagogically and cognitively comparable to human-vetted expert questions. The study seeks to determine the extent of this comparability across dimensions of quality, cognitive demand, and assessment validity.

2. Methodology

The study employs a mixed-methods pilot evaluation of an end-to-end AI-orchestrated workflow executed by a single human researcher coordinating multiple LLM agents.

A. Workflow Architecture

The research process was divided into six structured phases:

Data Identification: Retrieval of the official College Board SAT Math question bank (1,682 items) and OpenStax open-source textbooks.
Extraction & Corpus Construction: Conversion of PDFs to structured JSON/Markdown. The corpus was chunked into 2,980 pedagogically coherent segments (preserving definitions and examples) to serve as grounding context.
Alignment: Baseline MCQs were mapped to specific textbook chunks using vector embeddings (combining metadata and content queries) to ensure retrieval relevance.
Artifact Generation: Two LLMs (GPT-5-nano and Gemini-2.5-Flash) were prompted to generate new MCQs based on the retrieved textbook chunks and specific metadata constraints (domain, skill, difficulty, cognitive level).
Evaluation: A 24-criterion quality rubric (rated 1–5) was applied. The criteria covered six categories: Accuracy & Grounding, Clarity & Presentation, Distractor & Answer Quality, Difficulty & Engagement, Domain & Skill Alignment, and Difficulty & Cognitive Calibration.
Judging: Two independent LLMs (acting as judges) evaluated three sets of data: the original SAT baseline, Gemini-generated MCQs, and GPT-generated MCQs.

B. Statistical Analysis

To assess comparability, the authors utilized Equivalence Testing (TOST - Two One-Sided Tests) rather than traditional null-hypothesis significance testing.

Equivalence Bound: Set at $\delta = 0.2$ SD (small effect size).
Tracks: Analysis was conducted on both a "Matched" track (paired comparisons) and a "Whole-set" track.
Criteria for Equivalence: A criterion was deemed equivalent only if all pairwise comparisons (Gemini vs. SAT, GPT vs. SAT, Gemini vs. GPT) fell within the equivalence bounds.

3. Key Contributions

Empirical Workflow Documentation: Provides a concrete account of a single researcher orchestrating a complex, multi-agent pipeline for data extraction, generation, and evaluation, demonstrating a shift from content creation to system governance.
Large-Scale Comparative Evaluation: Conducted a rigorous evaluation of ~1,000 generated MCQs against a human baseline using a 24-criterion framework and dual LLM judges.
Identification of "Surface vs. Depth" Gaps: Demonstrated that while LLMs excel at surface-level quality, they fail to achieve equivalence in deep cognitive and calibration dimensions.
Definition of "AI Research Operations": Articulates a new set of core competencies for scientific work, including specification design, error control, provenance auditing, and orchestration.

4. Key Results

A. Aggregated Quality

High Average Scores: All sets achieved high mean scores (4.64–4.89/5).
Surprising Baseline Performance: Interestingly, LLM-generated MCQs received slightly higher average scores from LLM judges than the human-vetted SAT baseline. The authors attribute this to "stylistic alignment," where LLMs generate text that better matches the internal formatting and phrasing preferences of the judging models.

B. Criterion-Level Diagnostics

Strong Performance (Surface Level): Criteria related to grammar, fluency, lack of duplicates, factual accuracy, and option clarity achieved near-ceiling scores (>4.9). LLMs have effectively solved the "surface" aspects of MCQ authoring.
Weak Performance (Deep Level): Significant gaps were found in:
- Skill Depth: Questions often invoked skills superficially rather than probing them rigorously.
- Cognitive Engagement: Generated items failed to elicit the intended level of cognitive processing.
- Difficulty Calibration: Generated questions struggled to match the precise difficulty levels specified in the metadata.
- Distractor Plausibility: Distractors were often less plausible or less homogeneous than expert-written ones.

C. Equivalence Testing Outcomes

Strict Equivalence Failed: No set of generated MCQs achieved equivalence on all 24 criteria.
Partial Equivalence: Depending on the judge and analysis track, only 9 to 12 out of 24 criteria met the equivalence threshold.
Persistent Gaps: The criteria that consistently failed equivalence were concentrated in Skill Depth, Cognitive Engagement, Difficulty Calibration, and Metadata Alignment.

D. Work-Practice Findings (Labor Shift)

Time Compression: A project estimated to take a PhD student 6 months was completed in 10 days (approx. 80 hours) with only $62 in API costs.
Role Transformation: The researcher spent 52.5% of their time on Pipeline Engineering and Operations (building scripts, handling failures, managing tools) and 15% on Evaluation Design. Only 10% was spent on Human Quality Control/Interpretation.
No "Hands-Free" Automation: The work did not disappear; it shifted from doing the tasks to managing the system, defining constraints, and verifying outputs.

5. Significance and Implications

Redefining Scientific Labor: The study confirms that AI does not merely accelerate existing workflows but fundamentally alters the division of labor. The "scarce" resource in AI-empowered research is no longer content generation but orchestration, specification, and governance.
Emergence of "AI Research Operations": The authors propose that a new professional capability is emerging, combining domain expertise with software engineering and evaluation literacy (similar to the historical emergence of bioinformatics).
Limitations of LLM-as-a-Judge: The study highlights that LLM judges introduce systematic biases (e.g., favoring their own generation style). It argues for explicit governance, documentation of evaluator provenance, and the inclusion of human expert raters in future validation loops.
Feasibility vs. Validity: While AI can rapidly produce high-volume, surface-quality artifacts, achieving deep pedagogical validity (cognitive depth and precise calibration) remains a significant challenge requiring human oversight.

In conclusion, the paper argues that the future of scientific work lies not in replacing researchers with AI, but in equipping researchers with the skills to orchestrate AI systems to perform high-throughput research, shifting the human focus from execution to system-level control and verification.