💻 computer science

A simulation-based framework for sizing paired benchmarks in the evaluation of clinical artificial intelligence, applied to 168 expert-level dyslipidaemia items

This paper presents a simulation-based framework for determining the necessary sample size in paired clinical AI benchmark evaluations to address limitations of traditional methods, demonstrating through a dyslipidaemia study that pre-calculated bank size, rather than system performance alone, dictates whether comparative conclusions are statistically reachable.

Original authors: Mete Ucdal, Karya Yurtsever, Pınar Yıldız, Ayşen Akalın, Kadir Uğur Mert, Gülay Sain Güven

Published 2026-09-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mete Ucdal, Karya Yurtsever, Pınar Yıldız, Ayşen Akalın, Kadir Uğur Mert, Gülay Sain Güven

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of medical technology, a new generation of artificial intelligence systems is being trained to diagnose diseases and recommend treatments. These systems, often built on vast networks of data, are increasingly tested against human experts to see who performs better. The standard way to compare them is to give both the computer and the doctor the same set of medical questions and count the correct answers. However, a critical problem has emerged in this field: researchers often do not know how many questions are needed to make a fair comparison. If the test is too short, a smart computer might get every answer right simply by chance, making it impossible to tell if it is truly better than a human or just lucky. Conversely, if the difference between two systems is very small, a short test might miss it entirely, leading scientists to wrongly conclude that the systems are identical when they are not. Without a clear plan for how many questions to ask, the results of these high-stakes comparisons can be misleading, leaving hospitals and patients unsure of which technology is safe and effective to use.

A team of researchers set out to solve this problem of measurement by creating a new framework for designing these tests, then applying it to a complex medical challenge: the management of high cholesterol and related blood fat disorders. They did not just run a test; they first calculated exactly how many questions were required to detect specific differences in performance. Using a method that simulates thousands of potential test outcomes, they determined that to reliably spot a small but meaningful advantage in a system, they needed a bank of questions far larger than the dozens typically used in previous studies. They then built this larger test, consisting of 168 expert-level medical scenarios, and put it to the trial. The goal was to evaluate a new type of artificial intelligence that combines a powerful language model with a strict set of safety rules, checking to see if this combination actually improves accuracy and safety compared to the language model alone, other advanced computers, and practicing doctors.

The results of this carefully sized experiment revealed a clear hierarchy of performance. The new system, which uses a "neurosymbolic" approach to check its own work against a locked set of medical rules, answered 97 percent of the questions correctly. This was a significant improvement over its own underlying engine, which got 88 percent right, and it also outperformed the best individual doctors and other leading artificial intelligence models. The researchers were able to pinpoint exactly where this improvement came from. They found that the complex internal organization of the system, where different parts of the AI talk to each other, added very little value on its own. The real gain came from the symbolic verifier, the component that acts like a strict editor, checking the AI's reasoning against established medical rules. This rule-checking step was responsible for the vast majority of the accuracy boost, correcting errors that the unverified system would have made.

Beyond simple accuracy, the study also examined how the systems handled difficult or unclear medical cases where a single right answer might not exist. Here, the new system showed a distinct advantage in safety. When faced with these ambiguous situations, the full system with the rule-checker declined to give an answer 85 percent of the time, choosing instead to flag the case for human review. In contrast, the underlying engine without the rule-checker only hesitated 4 percent of the time, often rushing to give a potentially unsafe recommendation. This behavior demonstrated that the system's design successfully prioritized caution over confidence, a crucial trait for medical tools. The researchers also noted that the quality of the questions mattered; the system's advantage was most pronounced on the highest-quality, most clearly defined medical cases, suggesting that rule-based checks work best when the rules themselves are clear and unambiguous.

Perhaps the most important finding of the study was not just about which system won, but about how the test itself was designed. The researchers compared their new, large-scale results with a previous, smaller study they had conducted on the same system using only 24 questions. In that earlier, smaller test, the system and its underlying engine had both scored perfectly, making it impossible to tell them apart. The new, larger test proved that the earlier result was an artifact of the test being too short to reveal the differences. By calculating the required size in advance, the team ensured that their conclusions were robust. They also identified which comparisons were still too difficult to resolve with a human-sized test, noting that detecting very small differences in how the system's internal parts work would require a test bank of over a thousand questions, a scale that is currently beyond the reach of manual expert testing.

The study concludes that the future of evaluating medical artificial intelligence depends on rigorous planning rather than just running a quick comparison. The researchers argue that institutions testing these tools must calculate the necessary number of questions before they begin, ensuring the test is long enough to detect the differences that matter for patient safety. They found that while the new system is superior, its benefits are specific: it excels at applying strict rules to clear cases and knowing when to step back on uncertain ones. The study does not claim that this system is ready for immediate use in patient care, as no real patients were involved, but it provides a clear, evidence-based path forward. It shows that with the right size and design, we can move beyond guessing and start measuring the true capabilities and limitations of medical artificial intelligence with the precision it demands.

Technical Summary: A Simulation-Based Framework for Sizing Paired Benchmarks in Clinical AI Evaluation

Problem Statement
Comparative evaluations of clinical artificial intelligence (AI) increasingly utilize paired designs where competing systems answer a shared bank of expert-authored items, with performance compared via McNemar's test. Despite established sample-size methodology for this design, it is rarely applied in practice. Evaluations are typically built to the length of time expert respondents can complete (often a few dozen items), without calculating the item count required to detect specific contrasts.

This omission is critical due to two features of contemporary AI benchmarks that standard normal-approximation formulae fail to address:

Discreteness: The exact McNemar test is discrete. To achieve statistical significance (two-sided $P < 0.05$ ) with no discordance in the opposing direction, a minimum of six one-directional discordant items is required. No increase in bank size can lower this floor; it only increases the probability of reaching it.
Ceiling Constraints: In fields where systems are highly accurate, a short bank may allow a strong system to achieve a perfect score frequently. If a system scores perfectly, it cannot be ranked against others regardless of the number of discordant pairs, rendering the benchmark useless for differentiation.

Previous work by the authors on a 24-item dyslipidaemia benchmark illustrated these failures: the system and its backbone both scored perfectly, no comparisons reached significance, and component-level ablations were unestimable. The authors argue that such "null" results often reflect instrument limitations rather than system inertness.

Methodology
The study proposes a prospective, simulation-based framework to determine the required bank size before item authoring.

Sizing Framework:
- Simulation: Required bank size ( $n$ ) is derived via Monte Carlo simulation (40,000 replications) over the trinomial distribution of paired outcomes ( $p_b$ : system correct/comparator incorrect; $p_c$ : system incorrect/comparator correct; $p_{concordant}$ ).
- Constraints: The simulation applies the exact McNemar test to each replication to find the smallest $n$ yielding 80% power. It explicitly enforces the discreteness floor (requiring sufficient probability of reaching 6 discordant pairs) and the ceiling constraint (ensuring the probability of a perfect score for a high-accuracy system, e.g., 97%, falls below a threshold, such as 1%).
- Application: The framework was used to size a new benchmark for dyslipidaemia. The target was a net discordance gain of 7.5 percentage points. The simulation indicated 184 items were needed for 80% power, but the ceiling constraint (requiring 152 items to avoid saturation for a 97% accurate system) and practical recruitment limits led to a target of 160 analysed items.
Benchmark Instrument:
- Items: 214 single-best-answer items were authored across five reasoning domains (drug interactions, myopathy, familial dyslipidaemia, special populations, novel therapeutics).
- Selection: Items were retained only if three independent reference keys (one human-led, one AI-generated, one expert pharmacologist) agreed unanimously. This yielded 168 analysed items and 46 "abstention probe" items (where keys disagreed) to test safety/abstention mechanisms.
- Stratification: Items were stratified into Tier A (high quality: unanimous rater endorsement, complete evidence provenance, unanimous key agreement) and Tier B (failing one or more criteria) before scoring.
Systems Evaluated:
- NS-MML: A neurosymbolic multi-agent system for lipidology using a GPT-5.5 backbone, orchestrated via LangGraph, with a symbolic verifier and abstention controller.
- Ablation Arms: Three strictly nested configurations were tested: (1) Backbone only; (2) Backbone + Multi-agent orchestration (verifier disabled); (3) Full NS-MML (verifier and abstention active).
- Comparators: Two frontier LLMs (Claude Opus 4.8, Gemini 2.5 Pro) and six practising physicians.
- Protocol: All responders used identical prompts, temperature 0, and were evaluated on the same items. Reproducibility was assessed via five seeds.

Key Results

Framework Validation: The framework successfully predicted which contrasts would be resolvable.
- The total architectural contrast (NS-MML vs. Backbone) was resolved at 8.9 percentage points with 97.7% realized power.
- The orchestration-only contrast (Orchestration vs. Backbone) remained unresolved at 2.4 points with only 13.3% realized power, requiring approximately 1,051 items to resolve.
- Ceiling probability for a 97% accurate system dropped from 48.1% at 24 items to 0.6% at 168 items.
Performance:
- NS-MML achieved 97.0% accuracy (163/168), significantly outperforming its backbone (88.1%), Gemini 2.5 Pro (83.9%), and all six physicians (range 79.8–88.1%).
- The advantage was most pronounced in Tier A items (98.4% vs. 84.4% for the backbone).
Component Attribution:
- The accuracy gain was attributed primarily to symbolic verification (6.5 percentage point increment), not multi-agent orchestration (2.4 percentage point increment, non-significant).
- The symbolic verifier altered 15 answers, 13 of which were corrected to the right option.
Safety and Abstention:
- On the 46-item abstention probe, NS-MML declined to answer 84.8% of under-determined items, compared to 4.3% for the backbone.
- NS-MML committed zero hard-stop rule violations, whereas the backbone committed 19.6% and Gemini 2.5 Pro 23.9%.
Null Findings:
- The lack of significance for the orchestration-only contrast was identified as an underpowered result (13.3% power), not evidence of no effect.
- The interaction between item quality (Tier A vs. B) and architectural benefit was not statistically significant ( $P=0.104$ ), though point estimates favored Tier A.

Significance and Claims
The paper claims its primary contribution is methodological: a sizing framework that converts the design of clinical AI evaluations from an ad-hoc process into an arithmetic one.

From Non-Results to Measurements: The study demonstrates that the same architecture, when evaluated on a properly sized instrument (168 items) versus an undersized one (24 items), yields measurable component effects rather than perfect ties and unestimable ablations.
Design Rules: The authors propose four design rules for the field:
1. Size the bank to the specific contrast before authoring items.
2. Report realized power for every contrast, especially null ones.
3. Measure item quality prospectively and stratify on it.
4. Match evaluation configurations and endpoints to the mechanisms under test (e.g., using abstention endpoints for safety mechanisms).
Limitations of Current Literature: The paper argues that many existing "null" results in clinical AI are actually structural consequences of short banks (saturation and discreteness floors) rather than evidence of system inertness.
Modest Claims: The authors explicitly state that the study does not license the use of the system in patient care. The findings justify a prospective clinician-in-the-loop trial but do not constitute clinical deployment evidence. The study also acknowledges that the "orchestration-only" benefit remains an open question requiring a much larger bank (approx. 1,000+ items) to resolve.

In conclusion, the paper asserts that "bank size, not the system under test, determined which conclusions were reachable," and urges the field to justify bank sizes a priori and report realized power to distinguish between unresolvable contrasts and true null effects.

Technical Summary: A Simulation-Based Framework for Sizing Paired Benchmarks in Clinical AI Evaluation

More like this