📄 medicine

Expert validation of AI-generated clinical assessment items for physician assistant certification: a multi-dimensional rubric and dual- reliability method

This study presents and validates a multi-dimensional rubric and dual-reliability method for systematically assessing AI-generated Physician Assistant certification exam items, demonstrating that the AI-produced content meets expert quality standards and is indistinguishable from human-written questions in student performance and perception.

Original authors: I Made Agus Setiawan, Maria Yuliana, Bayu Aryoyudanta, Firdaus A Indradhirmaya, Haomin Hu, Dipu Patel, David C. Beck, Andi Saptono, Bambang Parmanto

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: I Made Agus Setiawan, Maria Yuliana, Bayu Aryoyudanta, Firdaus A Indradhirmaya, Haomin Hu, Dipu Patel, David C. Beck, Andi Saptono, Bambang Parmanto

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to be a doctor. You don't just want it to memorize facts; you want it to take a test, solve a mystery, and explain its reasoning just like a human medical student. This is the world of Generative AI in education. Think of AI as a super-fast, tireless writer that can churn out thousands of practice questions in the time it takes a human professor to write one. But here's the catch: in high-stakes exams like the one Physician Assistants (PAs) must pass to get their license, a bad question isn't just annoying; it can be dangerous. If the question is confusing or the answer is wrong, students might learn the wrong thing.

For years, we've had a strict system for checking human-written questions. A team of experts reads every single question to make sure it follows the rules, is clear, and tests the right skills. But nobody knew how to apply this same "expert inspection" to AI. If an AI writes a question, how do you know it's actually good? Is it just a fancy word salad, or is it a genuine test of medical knowledge? This paper steps into that gap, asking: Can we build a reliable checklist to validate AI-generated medical questions, and does the AI actually produce work that is as good as a human's?

The AI Question Factory and the Expert Inspectors

The researchers at the University of Pittsburgh decided to build a "factory" that uses AI to generate practice questions for the Physician Assistant National Certifying Examination (PANCE). But before they could let students use these questions, they needed a way to inspect the factory's output. They couldn't just trust the AI; they needed a human quality control team.

So, they created a new inspection tool, which is like a detailed scorecard with five specific categories. Imagine you are judging a baking competition. You wouldn't just say "good cake." You'd check:

Clinical-Educational Alignment: Does this question actually test what a doctor needs to know?
Scenario Completeness: Is the story (the patient's symptoms) complete, or did the baker forget the eggs?
Answer Option Quality: Are the wrong answers (distractors) tricky but fair, or are they obviously wrong?
Language and Clarity: Is the question easy to read, or is it a confusing mess?
Explanation Quality: If the student gets it wrong, does the explanation teach them why?

The team used this scorecard to check 55 AI-generated questions. They brought in six expert PA faculty members to act as the judges. The results were surprisingly good. The questions scored very high on the overall quality scale (an average of 0.9285 out of 1.0), which means the AI was successfully following the rules of good question writing. The "Question Track" (the first four categories) and the "Explanation Track" (the fifth category) both passed the threshold for being "strong."

The Great "Agreement" Mystery

Here is where the story gets a little twisty and fascinating. When the six experts looked at the questions, they mostly agreed: "Yes, this is good." In fact, they agreed on the quality of the questions about 88% of the time.

But when the researchers tried to use a standard math formula (called Fleiss' κ) to measure how much the experts agreed, the number came out incredibly low—almost zero (0.1201). Usually, a low number like that means the experts were fighting and couldn't agree. But they were agreeing!

The paper explains this as a "prevalence paradox." Think of it like a classroom where 99% of the students get an A. If you try to calculate how much the students agree on who got an A, the math gets weird because almost everyone is doing the same thing. The standard formula gets confused by this "too much agreement" and thinks it's just luck.

To fix this, the researchers used a different, more robust math tool called Gwet's AC1. This tool gave a high score (0.8653), confirming that the experts were actually in sync. The paper argues that when AI is doing a good job, we will see this pattern often: high agreement that looks like low agreement to old-school math. They suggest we need to use this new tool to avoid panicking when AI content is actually excellent.

The "Human vs. Robot" Blind Test

Finally, the researchers wanted to know if students could tell the difference. They set up a blind test with 10 medical students. The students answered 30 questions: 15 written by the AI and 15 written by humans. They didn't know which was which.

The results were a tie. The students got 70.0% of the AI questions right and 69.3% of the human questions right. Statistically, that is a dead heat. The AI questions were just as hard and just as clear as the human ones.

However, when the students tried to guess which questions were written by a robot, they got it wrong more often than not. They guessed correctly only 52.67% of the time (which is basically a coin flip). Interestingly, they had a bias: they tended to think the AI questions were written by humans, but they were better at spotting the human questions as human. This suggests the AI is getting very good at sounding like a person, but it's not perfect yet.

What This Means (and What It Doesn't)

The paper concludes that they have built a working "inspection tool" that can validate AI questions for medical exams. The AI they tested produced high-quality content that passed expert review and performed just as well as human-written questions in a small student test.

However, the authors are careful not to say this is a finished, perfect solution. The student test was small (only 10 people), so they call it "preliminary." They also note that the AI still struggles a bit with "Answer Option Quality"—sometimes the wrong answers aren't quite tricky enough. The tool they built is designed to be used for other medical exams too, like nursing or surgery boards, but it needs to be tested there first.

In short, the AI isn't ready to replace the teachers yet, but with this new "scorecard" to check its work, we might finally be able to let it help write the practice tests that prepare the next generation of doctors.

Technical Summary: Expert Validation of AI-Generated Clinical Assessment Items

Problem Statement
High-stakes certification exams, such as the Physician Assistant National Certifying Examination (PANCE), require vast banks of high-quality clinical practice questions. Traditional expert-driven item development is labor-intensive, slow to scale, and struggles to keep pace with evolving clinical guidelines. While Generative AI (specifically Large Language Models with Retrieval-Augmented Generation) offers a path to scalable content production, there is no established methodology for systematically validating AI-generated assessment content against the same rigorous quality standards expected of human-written items. Existing literature focuses on AI's ability to take exams or the deployment of AI tools, but lacks a reusable, standard-grounded instrument to certify the quality of AI-generated items before student use. Furthermore, when expert raters largely accept AI content, traditional inter-rater reliability metrics (like Fleiss' $\kappa$ ) often yield misleadingly low scores due to skewed rating distributions, creating a "prevalence paradox" that obscures genuine agreement.

Methodology
The study employed a two-round validation workflow mirroring standard human item-development processes, applied to a retrieval-augmented, agentic generation engine developed by the authors for PANCE preparation.

Validation Instrument: A five-dimensional rubric grounded in published NCCPA item-writing guidelines was developed. The dimensions are:
- Track 1 (Question Construction): Clinical-Educational Alignment, Scenario Completeness, Answer Option Quality, and Language and Clarity.
- Track 2 (Explanation Quality): Explanation Quality.
- Ratings used a 3-point ordinal scale (Poor, Acceptable, Excellent).
- A "Two-Track Quality Gate" separated question construction from explanation quality, allowing items to be approved for practice use even if explanations required revision, and vice versa.
- Diagnostic tags and free-text comments were used to identify specific flaws (e.g., multiple correct answers, missing clinical details).
Expert Validation (Round 1 & 2):
- Round 1 (Pilot): Four PA faculty reviewed 128 items using a binary judgment. This round identified recurring content issues (relevance, answer option validity, clarity) and informed the refinement of the instrument.
- Round 2 (Main Study): Six PA faculty reviewed 55 items across 14 of the 15 PANCE organ systems.
- Reliability Analysis: The study reported dual inter-rater reliability (IRR) indices: Fleiss' $\kappa$ (chance-corrected) and Gwet's AC1/AC2 (prevalence-robust). This approach was designed to distinguish genuine disagreement from the statistical artifact caused by high acceptance rates.
Student Perception Study (Preliminary):
- A blinded within-subject study with 10 PA students compared 15 AI-generated items against 15 matched human-written items.
- Tasks included rating clarity/realism, guessing the source (AI vs. Human), and answering the questions.

Key Results

Content Validity: The overall Scale-Level Content Validity Index (S-CVI/Ave) was 0.9285, exceeding the 0.90 threshold for strong validity. Both the Question Construction track (0.9356) and the Explanation track (0.9000) cleared the threshold. However, "Answer Option Quality" (Dimension 3) was the weakest dimension (S-CVI/Ave = 0.8545), indicating this as a primary target for engine refinement.
The Prevalence Paradox: A significant discrepancy was observed in reliability metrics. Observed agreement was high ( $P_o \approx 0.88$ ), yet Fleiss' $\kappa$ was low (0.1201), falling into the "poor" range. Conversely, Gwet's AC1 was high (0.8653). The authors attribute this to the "prevalence paradox": when raters overwhelmingly accept items, chance-corrected indices like $\kappa$ are deflated by the skewed distribution, while prevalence-robust indices like AC1 accurately reflect the high level of agreement. This pattern was consistent across both validation rounds.
Diagnostic Findings: The most frequent issues flagged by raters involved "Answer Option Integrity" (e.g., multiple defensible correct answers) and "Missing Critical Clinical Detail."
Student Perception (Preliminary, n=10):
- Knowledge Equivalence: AI and human items produced nearly identical answering accuracy (70.0% vs. 69.3%, $p=0.89$ ).
- Perceptual Equivalence: Students rated clarity and realism as statistically equivalent between sources.
- Source Detection: Students could not reliably distinguish AI from human items (52.67% overall accuracy, not significantly different from chance). However, an asymmetry was noted: students correctly identified human items 59.3% of the time but correctly identified AI items only 46.0% of the time (below chance), suggesting a response bias toward labeling items as "human."

Key Contributions

A Reusable Validation Instrument: The study introduces a five-dimensional rubric and a two-track quality gate specifically designed to validate AI-generated clinical assessment items against published item-writing standards. It is designed to be transferable to any blueprint-driven certification exam (e.g., NCLEX, USMLE) given the appropriate content blueprint and guidelines.
Methodological Insight on Reliability: The paper demonstrates that the "prevalence paradox" is a predictable feature of validating high-quality AI content. It argues that reporting dual reliability indices (Fleiss' $\kappa$ alongside Gwet's AC) is essential to distinguish genuine expert disagreement from statistical artifacts caused by high acceptance rates.
Empirical Evidence of Equivalence: Preliminary data suggests that AI-generated items, when validated, can achieve parity with human-written items in terms of difficulty, clarity, and realism, though specific refinements are needed for answer option construction.

Significance and Claims
The authors claim this work fills a critical gap in medical education by providing a standard-grounded method to certify AI content before deployment, addressing the ethical and pedagogical risks of unvalidated AI in high-stakes exam preparation. The study posits that the instrument's design—anchored in published standards and utilizing a prevalence-robust reliability approach—offers a practical framework for any program seeking to scale content production without compromising quality. The authors explicitly state that the instrument is not tied to PA certification but is adaptable to other health professions, provided the necessary domain-specific inputs (blueprint, knowledge base, guidelines) are supplied. The findings regarding the prevalence paradox serve as a cautionary note for future studies, urging researchers to avoid misinterpreting low $\kappa$ values as evidence of poor expert agreement when the underlying content is of high quality.

The AI Question Factory and the Expert Inspectors

The Great "Agreement" Mystery

The "Human vs. Robot" Blind Test

What This Means (and What It Doesn't)

Technical Summary: Expert Validation of AI-Generated Clinical Assessment Items

More like this