💬 NLP

Measuring Competency, Not Performance: Item-Aware Evaluation Across Medical Benchmarks

This paper introduces MedIRT, an Item Response Theory-based evaluation framework that outperforms traditional accuracy metrics by jointly modeling latent medical competency and item characteristics, thereby providing more valid, stable, and nuanced assessments of Large Language Models across diverse medical benchmarks.

Original authors: Zhimeng Luo, Lixin Wu, Adam Frisch, Daqing He

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhimeng Luo, Lixin Wu, Adam Frisch, Daqing He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to hire a new doctor. You have a stack of 1,100 medical exam questions ranging from "What color is the sky?" (easy) to "Diagnose this rare, complex genetic disorder based on three conflicting symptoms" (hard).

Currently, most people evaluate AI doctors by simply counting how many questions they got right. If Model A gets 70% right and Model B gets 70% right, we assume they are equally good.

The authors of this paper say: "That's a terrible way to hire a doctor."

Here is why, and how they fixed it, explained in simple terms.

1. The Problem: The "Counting" Trap

Imagine two students take a test.

Student A answers 20 questions correctly, but they are all questions like "What is 2+2?"
Student B answers 20 questions correctly, but they are all questions like "Solve this quantum physics equation."

If you just count the score, both students get a "20." But clearly, Student B is much smarter.

The paper argues that current AI rankings are like Student A's score. They treat a "hard" medical question the same as an "easy" one. This leads to rankings that change depending on which specific test you use. It's like judging a chef only on how well they can boil water; if you switch to a test about baking, the ranking changes completely.

2. The Solution: MEDIRT (The "Smart Grader")

The authors created a new system called MEDIRT. Instead of just counting right answers, they use a method called Item Response Theory (IRT).

Think of IRT as a smart grader that understands the nature of the questions.

It knows that getting a "hard" question right is worth more points than getting an "easy" one.
It knows that some questions are "discriminators"—they are great at telling the difference between a genius and an average student.
It calculates a true ability score (like a hidden talent level) rather than just a raw score.

The Analogy:

Old Way (Accuracy): Counting how many rungs of a ladder a person climbed.
New Way (MEDIRT): Measuring how high they jumped, taking into account that some rungs are slippery and some are made of gold.

3. The "Quality Control" Check (EFA)

Before they even start grading, the authors did something crucial. They checked if the questions in each category (like "Cardiology" or "Psychology") actually made sense as a group.

Sometimes, a "Cardiology" test might accidentally include questions about "History of Medicine" or "How to write a prescription." If you mix those up, the test is broken.

The authors used a statistical filter (called EFA) to throw out the "bad apples" (confusing or unrelated questions) before grading the AI.
Result: They ended up with a cleaner, fairer test where every question in the "Cardiology" section was actually about the heart.

4. What They Found (The Surprises)

They tested 71 different AI models on this new, smarter system. Here is what they discovered:

The "Spiky" Geniuses:
Under the old system, one AI (GPT-5) looked like the clear winner. But under MEDIRT, they found that AI models are like specialist athletes.
- One model might be a world-class "Cardiologist" but terrible at "Psychology."
- Another might be average overall but a "Communication Expert" (great at talking to patients).
- Takeaway: There is no single "best" AI. You need to pick the right tool for the specific job.
The "Cheaters" (Difficulty-Insensitive Responding):
They found some models that were weird. These models would get easy questions wrong but somehow get harder questions right.
- Analogy: Imagine a student who fails a spelling test on "cat" and "dog," but somehow passes a test on "photosynthesis."
- The authors realized these models aren't actually "smart"; they are just guessing based on patterns or getting lucky with the format of the question. The old system would have praised them; the new system flagged them as risky for real-world use.
Better Predictions:
The new system was much better at predicting how an AI would perform on new questions it had never seen before. It was like a coach who could predict a player's performance in a new game, whereas the old system just looked at their past stats.

5. Why This Matters

If you are a hospital or a company trying to use AI for medicine, you can't just look at a leaderboard and pick the top number.

Old Way: "Model X has 75% accuracy. Hire it!" (Risk: It might fail on hard, real-world cases).
New Way (MEDIRT): "Model X is great at general knowledge but terrible at safety. Model Y is a specialist in heart disease but bad at communication. Choose based on what you need."

Summary

This paper is like upgrading from a cash register (which just counts money) to a financial analyst (which understands the value of the assets).

They built a better way to test medical AIs that:

Weighs hard questions more than easy ones.
Cleans up the test questions to make sure they are fair.
Reveals that AI models have "spiky" strengths and weaknesses, not just a single "smartness" score.
Catches models that are "faking it" by guessing patterns instead of knowing the answer.

It's a move from asking "How many did you get right?" to "What do you actually know, and where do you fail?"

1. Problem Statement

The current paradigm for evaluating Large Language Models (LLMs) in medicine relies heavily on aggregate accuracy (the percentage of questions answered correctly). The authors argue this metric is fundamentally flawed for three reasons:

Equal Weighting of Items: It treats all questions as equally informative, failing to distinguish between a model answering 20 easy questions correctly versus 20 difficult, high-discrimination questions.
Lack of Comparability: Without a unified measurement scale, scores from different benchmarks (e.g., MedQA vs. MedXpertQA) are incomparable.
Construct Validity Issues: Accuracy provides no mechanism to verify if a set of questions grouped under a "topic" actually measures a single, coherent underlying ability. It risks conflating distinct skills into a single score.

The paper posits that evaluation must shift from measuring performance (how many questions were right) to measuring competency (what the model actually knows and where it fails), requiring a framework that accounts for item difficulty and discrimination.

2. Methodology: The MEDIRT Framework

The authors introduce MEDIRT, a psychometric evaluation framework grounded in Item Response Theory (IRT). The framework consists of four integrated stages:

A. Benchmark Construction (USMLE-Aligned)

Data Source: Integrated 7,906 questions from three established benchmarks (MedQA, MedMCQA, MedXpertQA).
Topic Alignment: Questions were classified into 11 medical topics based on USMLE Step 1 Test Content Specifications (e.g., Cardiology, Neurology, Social Sciences) using GPT-oss-120b.
Stratification: A balanced dataset of 1,100 items (100 per topic) was created to prevent score distortion from topic imbalance.

B. Benchmark Integrity Validation (EFA)

Before modeling, the authors validated unidimensionality within each topic using Exploratory Factor Analysis (EFA).

Goal: Ensure that items within a topic measure a single coherent latent ability.
Process: Items with low factor loadings (<0.30) or negative loadings were pruned.
Result: The final dataset retained between 31 and 69 items per topic (645 total), ensuring that topic scores reflect a true latent construct rather than a heterogeneous mix of skills.

C. Large-Scale Inference

Cohort: 71 diverse LLMs (ranging from 3B to 400B+ parameters, proprietary and open-source) were evaluated via a standardized API protocol (OpenRouter).
Protocol: Zero-shot, multiple-choice questions with strict output formatting to ensure consistency.

D. Psychometric Modeling (2PL IRT)

Instead of a single multidimensional model, the authors built 11 independent unidimensional 2-Parameter Logistic (2PL) IRT models (one per topic).

Parameters Estimated:
- $\theta$ (Ability): The latent competency of the model on a specific topic.
- $b$ (Difficulty): The difficulty level of the item.
- $a$ (Discrimination): How well the item distinguishes between high and low-ability models.
Scoring: Models receive topic-specific ability scores ( $\theta_{m,t}$ ) and a composite score, calibrated for item characteristics rather than raw counts.

3. Key Contributions

MEDIRT Framework: A novel methodology integrating EFA-based benchmark integrity screening with topic-wise 2PL IRT modeling to provide stable, interpretable rankings.
Prospective Data Collection: Evaluation of 71 diverse LLMs on fresh data, avoiding the archival data limitations of prior work.
Dual Validation:
- Internal: Validated via hold-out prediction accuracy.
- External: Validated against 10 independent external benchmarks (expert preferences, safety, holistic tasks).
Diagnostic Depth: The framework identifies specific response patterns (Difficulty-Sensitive vs. Difficulty-Insensitive) and domain-specific "spiky" competency profiles that aggregate scores hide.

4. Key Results

A. Superior Generalization (Internal & External)

Internal Validation: MEDIRT achieved 83.3% accuracy in predicting held-out LLM responses on unseen questions, outperforming the model trained on the unpruned dataset (81.2%). This confirms that EFA pruning removes noise and yields a more stable scale.
External Validation: IRT-based rankings outperformed accuracy-based rankings across 6 independent external benchmarks (including expert preferences and safety judgments).
- Performance: 4 wins, 0 losses, 2 ties against accuracy.
- Stability: IRT rankings showed 18% lower variance across benchmarks, indicating greater robustness.
- Correlation: IRT showed stronger alignment with expert preference and safety judgments, while accuracy rankings were often artifacts of the specific item pool.

B. "Spiky" Competency Profiles

Aggregate rankings mask significant domain-specific heterogeneity.

Example: While GPT-5 was the overall leader, Kimi-k2 (ranked 12th overall) outperformed GPT-5 by nearly one standard deviation in Communication/Interpersonal Skills, despite severe deficits in Respiratory/Renal systems.
Implication: A model ranked 12th overall may be the optimal choice for a specific clinical workflow requiring its unique strength.

C. Divergent Response Profiles (DSR vs. DIR)

The study identified two distinct response behaviors that aggregate accuracy cannot distinguish:

Difficulty-Sensitive Responding (DSR): The majority of models. Accuracy declines monotonically as item difficulty increases (e.g., Mistral-7b). This aligns with expected knowledge hierarchies.
Difficulty-Insensitive Responding (DIR): A subset of models (including some top-tier ones like Codex Mini) where accuracy inverts (e.g., performing worse on easy items than medium ones).
- Cause: Likely driven by surface-format familiarity or fluency misattribution rather than deep knowledge.
- Risk: These models appear competent in aggregate but exhibit "brittle" behavior that poses deployment risks.

5. Significance and Implications

Shift in Evaluation Paradigm: The paper argues that simple accuracy is insufficient for high-stakes domains like medicine. MEDIRT transforms evaluation from a "ranking exercise" into a diagnostic instrument.
Clinical Deployment: The framework helps practitioners make informed decisions:
- Topic Selection: Choose models based on specific domain strengths (e.g., using a specialist model for communication tasks).
- Safety Checks: Detect DIR models that may fail unpredictably in real-world scenarios despite high aggregate scores.
Generalizability: While focused on medicine, the methodology (EFA validation + IRT modeling) is applicable to any high-stakes domain where benchmark integrity can be validated and items vary in difficulty.

In conclusion, MEDIRT demonstrates that by modeling the interaction between model ability and item characteristics, we can obtain a more valid, stable, and actionable understanding of LLM capabilities in medicine, moving beyond the "black box" of aggregate accuracy.