MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge

Technical Summary of MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge

Problem Statement

While Large Language Models (LLMs) are increasingly utilized in MRI research for protocol design, parameter selection, and troubleshooting, their reliability in domain-specific roles remains unverified. Existing MRI benchmarks, such as those based on 1995 technologist review books, have reached performance ceilings for top proprietary models (e.g., o1 Preview scoring 94%), offering limited discriminative power. Crucially, no systematic evaluation has tested LLMs on vendor-specific operational content—central to daily research MRI work—which includes specific protocol parameters, pulse sequence behaviors, and console operations. Furthermore, high Multiple-Choice Question (MCQ) scores may reflect option recognition rather than genuine generative recall, a distinction untested in prior MRI-specific evaluations.

Methodology

The study introduces MRI-Eval, a tiered benchmark comprising 1,365 scored items (1,355 MCQs and 10 misconception questions) across nine technical categories and three difficulty tiers.

• Data Sources: Items were drawn from authoritative textbooks, GE SIGNA Works scanner application manuals, GE EPIC Programming Course materials, and 35 expert-generated questions addressing knowledge gaps.
• Categories: The nine categories include GE scanner operations/protocol (n=406), pulse sequences (n=330), GE domain knowledge (n=155), safety, k-space/image formation, artifacts, T1/T2 relaxation, SNR/image quality, and parallel imaging.
• Difficulty Tiers: Tier 1 (83.1%) tests single-concept recall; Tier 2 (15.6%) requires cross-concept integration; Tier 3 (1.3%) poses expert-integration scenarios (noted as a limitation due to small sample size).
• Models Evaluated: Five models from distinct families were tested: GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6, Gemini 2.5 Pro, and Llama 3.3 70B.
• Evaluation Conditions:
  1. Primary MCQ: Zero-shot, four-option format with shuffled answers.
  2. Stem-Only (Free-Text): Options removed; models generated free-text responses scored by an independent LLM judge (Grok). Gemini 2.5 Pro was excluded here due to mandatory reasoning tokens affecting formatting.
  3. Primed Diagnostic: Models were presented with an incorrect answer claim and asked to agree/disagree, testing for sycophancy or retrieval assistance.
• Scoring & Analysis: Statistical analysis included Wilson score confidence intervals, McNemar's tests for pairwise comparisons, and Cohen's kappa for repeatability. A post-hoc audit revealed pervasive option-length cues (correct options were >1.3x the mean distractor length in 84.9% of items).

Key Contributions

1. Tiered Benchmark Construction: A specialized dataset incorporating underrepresented vendor-specific (GE) operational content, designed to locate knowledge boundaries rather than merely confirm competence.
2. Complementary Evaluation Formats: The inclusion of a stem-only condition alongside MCQs to quantify the gap between option recognition and generative recall, alongside a primed diagnostic condition.
3. Rigorous Model Comparison: A five-model comparison with pairwise statistical testing and run-to-run repeatability assessment (Cohen's kappa 0.983–0.998 for tested models).
4. Reproducibility: Public release of the evaluation harness and analysis scripts, while withholding the full question bank to prevent training-data contamination.

Results

• MCQ Performance: Overall accuracy was high and compressed, ranging from 93.2% (Llama 3.3 70B) to 97.1% (GPT-5.4). The spread among the four frontier models was only 2.4 percentage points.
• Category Variance: GE scanner operations was the lowest-scoring category for all models (88.2%–94.6%), while physics fundamentals (e.g., T1/T2 relaxation) approached ceiling performance.
• Stem-Only Degradation: Removing options caused a dramatic drop in accuracy for all models. Frontier models fell to 58.4%–61.1% (a 35.5–36.8 pp drop), while Llama 3.3 70B dropped to 37.1% (a 56.1 pp drop).
• Vendor-Specific Recall: In the stem-only condition, accuracy for GE scanner operations collapsed to 13.8%–29.8%, indicating that near-ceiling MCQ performance did not translate to strong free-text recall for vendor-specific content.
• Primed Condition: Incorrect answer suggestions generally increased accuracy (answer-cueing effect), though Llama 3.3 70B showed reduced accuracy at higher difficulty tiers under priming, suggesting weaker models may accept incorrect primes when knowledge is insufficient.
• Repeatability: Run-to-run repeatability was near-perfect across models, with disagreements concentrating in the GE scanner operations category.

Significance and Claims

The paper argues that strong MCQ performance on MRI content can mask weak generative recall, particularly for vendor-specific operational knowledge. The benchmark serves primarily as a relative model-comparison tool rather than an absolute measure of competency or deployment readiness.

Key conclusions include:

• Limitations of MCQs: High MCQ scores in specialized domains may reflect option recognition and length cues rather than deep understanding.
• Operational Risk: The sharp decline in stem-only performance for GE-specific content supports caution against relying on raw LLM outputs for vendor-specific protocol guidance in research settings.
• Future Directions: The findings motivate the development of retrieval-augmented or document-grounded systems as a necessary next step, as pure parametric knowledge appears insufficient for operational tasks.
• Evaluation Standards: Future domain-specific benchmarks should incorporate open-ended or stem-only formats to better distinguish between models, as standard MCQs compress performance near the ceiling.

The authors emphasize that these findings are based on controlled model-to-model comparisons and do not constitute a calibration against human expert performance or a direct estimate of clinical deployment readiness.