Technical Summary: MEDMARKS

Problem Statement

Evaluating Large Language Models (LLMs) for medical applications faces three primary challenges: benchmark saturation, limited data accessibility, and insufficient coverage of relevant clinical tasks. Existing suites like MultiMedQA have largely saturated due to their reliance on basic knowledge recall tasks that frontier models have mastered. Other benchmarks, such as HealthBench, focus narrowly on patient-facing conversations, while suites like MedHELM rely heavily on restricted or proprietary datasets that prevent community replication. Furthermore, many existing evaluations fail to reflect real-world use cases, such as generating treatment plans from medical reports or engaging in open-ended clinical reasoning. There is a critical need for a regularly updated, fully open-source, and reproducible evaluation suite capable of benchmarking a wide spectrum of models across diverse, clinically relevant tasks.

Methodology

The authors introduce MEDMARKS, a comprehensive, fully open-source evaluation suite comprising 30 benchmarks. The suite is organized into two primary subsets:

1. MEDMARKS-V (Verifiable): Contains multiple-choice question answering (MCQA) and other tasks with deterministic ground truths (e.g., medical calculations, information extraction).
2. MEDMARKS-OE (Open-Ended): Contains non-verifiable tasks such as patient-facing conversations, diagnostic reasoning, and note generation, evaluated using LLM-as-a-Judge.

Benchmark Selection and Implementation

The 30 benchmarks span a variety of tasks including:

• Knowledge & Reasoning: MedQA, MedMCQA, PubMedQA, MedXpertQA (expert-level), and MetaMedQA (metacognition).
• Clinical Tasks: MedCalc-Bench (calculations), MEDEC (error correction), MedDialog (summarization), and AgentClinic/MedAgentBench (multi-turn agentic interactions).
• Open-Ended Evaluation: HealthBench, MedicationQA, and MedCaseReasoning.

To ensure reproducibility, all benchmarks are implemented as verifiers environments (Brown, 2025) with corresponding reward functions. A subset of these datasets, termed MEDMARKS-T, includes explicit train/test splits, enabling their direct use as reinforcement learning (RL) environments for post-training.

Evaluation Protocol

• Models Evaluated: The study evaluated 61 models across 71 configurations, including proprietary frontier APIs (e.g., GPT-5.1/5.2, Gemini 3 Pro, Grok 4, Claude Sonnet 4.5) and open-source models (e.g., Qwen3, Baichuan-M3, GLM-4, Llama 3).
• Grading Mechanisms:
  • MCQA: A robust grading function was developed to handle formatting variations, stripping thinking traces, normalizing capitalization/punctuation, and accounting for negation to prevent false negatives.
  • Open-Ended: The authors utilized LLM-as-a-Judge with a panel of judges (primarily GPT-5 mini, paired with Grok 4.1 Fast or Gemini 3 Flash Preview) to evaluate semantic similarity and adherence to rubrics, avoiding the limitations of lexical overlap metrics like ROUGE.
• Metrics: Models are ranked using a weighted mean win rate, aggregating performance across datasets with log-size weighting to balance dataset influence.

Key Contributions

1. Comprehensive Open-Source Suite: MEDMARKS is presented as the largest completely open-source automated evaluation suite for medical capabilities, covering 30 benchmarks across verifiable, open-ended, and agentic tasks.
2. Scalable Evaluation: The authors performed a systematic evaluation of 61 models (including 6 medical-specific fine-tuned models) across 71 configurations, providing the most extensive publicly documented comparison of medical LLM capabilities to date.
3. RL-Ready Environments: By implementing benchmarks as verifiers with reward functions, the suite facilitates medical-specific post-training. The MEDMARKS-T subset allows researchers to use these datasets for Reinforcement Learning with Verifiable Rewards (RLVR).
4. Reproducibility: All code and benchmarking infrastructure are open-sourced, and the suite is designed as a "living leaderboard" to track the frontier of medical LLM capabilities as new models are released.

Key Results

The evaluation yielded several significant findings regarding the current state of medical LLMs:

• Performance Leaders: Frontier reasoning models (specifically Gemini 3 Pro Preview, GPT-5.1, and GPT-5.2) achieved the highest performance across both verifiable and open-ended benchmarks.
• Token Efficiency: While frontier proprietary models generally outperformed open-weight alternatives, they were also significantly more token-efficient. Open-source models often required 5x or more tokens to approach the performance of frontier models, highlighting a substantial optimization gap.
• Medical Fine-Tuning: Medically fine-tuned models (e.g., Baichuan-M2, MedGemma) generally outperformed their generalist counterparts, demonstrating the value of domain-specific post-training. However, some medical models underperformed newer generalist reasoning models on specific tasks.
• Reasoning vs. Instruction: Generally, "reasoning" variants of models outperformed their "instruct" counterparts. However, this trend was not universal; the Ministral family saw reasoning models underperform their instruction-tuned versions.
• Difficulty Gradient: Benchmarks span a wide difficulty range. Simple long-context tasks (e.g., PubHealthBench) were solved with high accuracy (>0.80), while expert-level clinical reasoning tasks like MedXpertQA remained largely unsolved (scores ~0.23), indicating a significant gap in current capabilities for high-stakes clinical reasoning.
• Bias and Robustness:
  • Answer-Order Bias: Models, particularly smaller ones and Grok 4, exhibited susceptibility to answer-order bias in MCQA tasks.
  • Overthinking: Models tended to generate more tokens when answering incorrectly, suggesting they struggle to recognize when they do not know an answer.
  • Quantization: 8-bit quantization was largely benign, but 4-bit quantization introduced consistent performance penalties.
• Tool Usage: When provided with calculator tools for medical calculations, many models either failed to use them, ignored correct tool outputs, or generated malformed tool calls, leading to performance regressions in some cases.

Significance and Claims

The paper claims that MEDMARKS addresses the critical need for a transparent, reproducible, and comprehensive benchmark suite that moves beyond simple knowledge recall to assess real-world medical reasoning and safety. By providing a "living leaderboard" and RL-ready environments, the authors aim to accelerate the development of safer and more capable medical AI.

The authors explicitly acknowledge limitations: the suite is still heavily biased toward single-turn question answering, lacks extensive evaluation of fairness/bias and safety beyond hallucination detection, and focuses solely on text-only tasks (excluding multimodal capabilities). They state that benchmark performance is not equivalent to clinical competence and must be interpreted with care, emphasizing that MEDMARKS is a tool for analysis and improvement rather than a definitive certification of clinical readiness.