🤖 AI

Fully Open Meditron: An Auditable Pipeline for Clinical LLMs

This paper introduces Fully Open Meditron, the first end-to-end auditable pipeline for clinical LLMs that combines a clinician-verified training corpus and reproducible framework to achieve state-of-the-art performance on medical benchmarks while maintaining full transparency and reproducibility.

Original authors: Xavier Theimer-Lienhard, Mushtaha El-Amin, Fay Elhassan, Sahaj Vaidya, Victor Cartier-Negadi, David Sasu, Lars Klein, Mary-Anne Hartley

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xavier Theimer-Lienhard, Mushtaha El-Amin, Fay Elhassan, Sahaj Vaidya, Victor Cartier-Negadi, David Sasu, Lars Klein, Mary-Anne Hartley

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart student who has read almost every book in the world. This student is great at answering general questions, but they haven't studied medicine yet. You want to turn them into a top-tier medical expert.

For a long time, the way to do this was to take this student, feed them a secret pile of medical textbooks and case studies, and then say, "Here are the answers you need to memorize." But there was a catch: nobody was allowed to see the textbooks or the study notes. You only got to see the final exam results. This meant you couldn't verify if the student actually learned the right things, or if they just memorized the answers to the specific questions they were tested on.

This paper introduces a new approach called Fully Open Meditron. Think of it as opening the entire classroom door, the library, and the teacher's lesson plans for everyone to inspect.

Here is the breakdown of their "Fully Open" recipe:

1. The "Open Kitchen" Philosophy

Most "open" medical AI models are like restaurants that only show you the final dish (the model weights) but keep the recipe, the ingredients, and the chef's notes hidden.

The Problem: If you can't see the ingredients, you don't know if the chef used fresh, verified medical guidelines or if they just memorized the menu from a previous exam.
The Solution: The authors built a pipeline where everything is public. They released the base model, the exact data they fed it, the code they used to train it, and the rules they followed. It's like a restaurant where you can walk into the kitchen, watch the chef cook, and taste the raw ingredients.

2. Building the "Study Library" (The Corpus)

To train these models, they didn't just grab random medical questions from the internet. They built a massive, organized library with three distinct sections:

The Exam Hall: They gathered eight existing public medical question banks (like practice tests for doctors) and cleaned them up so they all spoke the same language.
The Guideline Library: They took 46,469 real-world clinical practice guidelines (the official rulebooks doctors follow) and turned them into question-and-answer pairs. This ensures the AI learns from the actual rules of medicine, not just old exam questions.
The "What-If" Scenarios: They created new, complex patient stories (vignettes) that mimic real emergency room situations. They used a "teacher" AI to generate these, but then had four real doctors check the work to make sure the stories were realistic and the answers were correct.

Why this matters: Previous medical AI libraries were missing a lot of "emergency" and "life-threatening" cases. This new library is like a training simulation that specifically fills in those dangerous gaps, ensuring the AI is ready for the worst-case scenarios, not just routine check-ups.

3. The "Clean Room" (Decontamination)

A major issue in AI is "cheating." If the AI has seen the test questions before during its training, it's not actually smart; it's just memorizing.

The Fix: The authors ran a rigorous "decontamination" process. They scanned their entire training library against all the standard medical tests used to grade AI. If they found even a tiny overlap (like a shared phrase or sentence), they threw that data out. This ensures the AI is learning the concepts, not just the answers.

4. The "Final Exam" (Evaluation)

How do you test a medical AI? Usually, you give it multiple-choice questions (MCQs). But the authors argue this is like testing a pilot only on a simulator with fixed paths. Real medicine is messy and open-ended.

The New Test: They created a new evaluation method called Auto-MOOVE. Instead of just checking if the AI picked "A, B, C, or D," they ask the AI to write out its reasoning for complex patient cases.
The Judge: They used a powerful AI to grade these answers, but they first calibrated this AI judge against 204 human doctors to make sure the AI judge was fair and accurate. They also tested the models on HealthBench, a benchmark where doctors write detailed rubrics for what a "good" answer looks like.

The Results: Did it Work?

They applied this "Fully Open" recipe to five different base models. The results were impressive:

Better than the Base: Every single model trained with this open recipe became significantly better at medical tasks than its original, untrained version.
Beating the "Secret" Models: One of their models, built entirely from open data and open code, performed better than MedGemma, a famous medical model that uses secret, proprietary data.
New Record: Their largest model (Apertus-70B-MeditronFO) set a new record for the best performance ever achieved by a fully open medical AI.

The Bottom Line

The paper claims that you don't need secret, proprietary data to build a world-class medical AI. By being completely transparent—sharing the data, the code, and the training process—you can build a model that is not only highly accurate but also auditable. Doctors and regulators can look under the hood to see exactly how the AI learned, ensuring it's safe and reliable.

In short: They proved that if you build a medical AI with a clear, open, and rigorous recipe, it can perform just as well as (and sometimes better than) those built with secret ingredients.

Technical Summary: Fully Open Meditron: An Auditable Pipeline for Clinical LLMs

Problem Statement

Current Large Language Model (LLM) based Clinical Decision Support Systems (LLM-CDSS) suffer from a fundamental lack of transparency. While many "open" medical models release weights, they withhold the data provenance, curation procedures, and generation pipelines that dictate model behavior. This opacity prevents rigorous auditing of whether models learned from guideline-grounded evidence, benchmark artifacts, or synthetic hallucinations. Furthermore, standard medical benchmarks are often saturated, leading to performance gains that may reflect data contamination or memorization rather than genuine clinical capability. Existing "Fully Open" (FO) models, which expose the complete training stack end-to-end, do not currently exist in the medical domain, leaving a gap between the need for auditable AI and the reality of opaque adaptation pipelines.

Methodology

The authors introduce Fully Open Meditron, the first fully open pipeline for adapting FO foundation models into medical specialists. The methodology consists of three core components:

1. The Fully Open Meditron Corpus

The corpus is constructed to address distributional gaps in public medical datasets (e.g., underrepresentation of emergency care, life-threatening cases, and low-resource settings).

Aggregation: Eight public medical QA datasets (MedQA, MedMCQA, PubMedQA, MedExpQA, HealthSearchQA, LiveQA, AfriMed-QA v1/v2) are unified into a normalized conversational format with step-by-step rationales.
Synthetic Expansion: To expand coverage, the authors employ GPT-OSS-120B to generate three synthetic components, validated by a panel of four physicians:
- Synthetic Curated QA: Novel exam-style QA pairs seeded from the curated pool, stratified by question type.
- Guidelines QA: QA pairs grounded in 46,469 clinical practice guidelines from 16 global institutions.
- Synthetic MOOVE: Open-ended clinical vignettes seeded from the MOOVE training split to elicit complex diagnostic reasoning.
Quality Control: The pipeline enforces system-wide decontamination using a two-stage n-gram and token-alignment pipeline (adapted from Apertus) to eliminate overlap with evaluation benchmarks. Synthetic targets undergo gold-label rejection sampling (up to 8x at T=0.7) to mitigate hallucinations.

2. Training Framework

The recipe is applied to five fully open base models: Apertus-70B/8B-Instruct, OLMo-2-32B-SFT, EuroLLM-22B/9B-Instruct, and one open-weight control, Gemma-3-27B-IT.

Models are adapted via supervised fine-tuning (SFT) on the Fully Open Meditron corpus.
Training preserves the native instruction-tuning format of each base model.
Infrastructure utilizes NVIDIA GH200 nodes with Flash Attention 2 and mixed-precision training.

3. Evaluation Protocol

The authors move beyond standard multiple-choice questions (MCQs) to capture open-ended clinical reasoning.

Structured Benchmarks: Evaluation on MedQA, MedMCQA, PubMedQA, and the held-out MedXpertQA (for OOD checks).
Open-Ended Evaluation:
- HealthBench: Uses physician-authored rubrics to score open-ended responses.
- Auto-MOOVE: An automated "LLM-as-a-judge" protocol using Qwen3-235B-A22B (validated against 204 human raters) to perform pairwise comparisons on clinical vignettes. It assesses nine criteria: question comprehension, logical reasoning, relevance, harmlessness, fairness, contextual awareness, communication, clarity, and guideline alignment.

Key Contributions

A Fully Open Medical Adaptation Framework: A reproducible, end-to-end pipeline spanning corpus construction, synthetic generation, decontamination, training, and evaluation.
A Clinician-Audited Knowledge Corpus: A unified dataset combining public QA with synthetic extensions that significantly shift coverage toward emergency care (15.0% $\to$ 38.7%) and life-threatening severity (8.6% $\to$ 31.8%).
An Automated Open-Ended Evaluation Protocol: The Auto-MOOVE framework, validated against human inter-rater agreement, to assess multidimensional clinical reasoning.
A Family of Fully Open Medical Specialists: Five new models demonstrating that competitive medical specialization is achievable under FO constraints.

Results

Benchmark Performance: All MeditronFO variants outperform their respective base models. Apertus-70B-MeditronFO establishes a new state-of-the-art (SoTA) for fully open models, improving aggregate medical benchmark scores from 47.2% to 53.8% (+6.6 points).
Comparison with Proprietary/Open-Weight Models:
- Apertus-70B-MeditronFO narrows the gap to MedGemma-27B (60.67% vs 53.77%) but remains the strongest fully open system.
- Gemma-3-27B-MeditronFO is preferred over the proprietary MedGemma in 58.6% of Auto-MOOVE pairwise comparisons and outperforms it on HealthBench (58.0% vs 55.9%).
Open-Ended Evaluation: Every MeditronFO variant is preferred over its base in pairwise evaluations (win rates ranging from 67.2% to 92%). Improvements are consistent across all nine clinical criteria, not limited to a single dimension.
Ablation Studies: Removing specific corpus components reveals trade-offs. Removing "Guidelines QA" slightly improved MCQA accuracy but left open-ended preference unchanged, while removing "Curated QA" significantly degraded open-ended quality, indicating that exam-style supervision contributes to broader clinical reasoning.

Significance and Claims

The paper claims that Fully Open Meditron demonstrates that fully open pipelines can achieve state-of-the-art domain-specific performance without sacrificing auditability or reproducibility. The authors argue that the performance gap between open and closed models is a result of corpus construction rather than an inherent limitation of open models. By providing a transparent, clinician-audited pipeline, the work enables rigorous validation and addresses the "opacity" problem in medical AI. The results suggest that clinically competitive specialization can be achieved through disciplined data curation and open-ended evaluation, offering a reusable foundation for future fully open medical models.

The authors remain modest regarding limitations, noting that their decontamination is syntactic rather than semantic, that synthetic data accounts for ~64% of the corpus (with auditing limited to sampled QA pairs), and that the single teacher/judge setup may introduce stylistic biases. They emphasize that the open nature of the corpus is a partial mitigation for risks (enabling auditing) but also a partial amplifier (reproducibility by parties who may not audit).