💻 computer science

DALPHIN: Benchmarking Digital Pathology AI Copilots Against Pathologists on an Open Multicentric Dataset

The paper introduces DALPHIN, the first open multicentric benchmark for digital pathology AI copilots, which evaluates general-purpose and pathology-specific models against 31 pathologists across 130 diagnoses and finds that PathChat+ achieves expert-level performance in four out of six tasks.

Original authors: Carlijn Lems, Sander Moonemans, Natálie Klubíčková, Biagio Brattoli, Taebum Lee, Seokhwi Kim, Veronica Vilaplana, Laura Pons, Sapir Hochman, Mauricio Eduardo Suárez-Franck, Pedro Luis Fernandez, Juliu

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Carlijn Lems, Sander Moonemans, Natálie Klubíčková, Biagio Brattoli, Taebum Lee, Seokhwi Kim, Veronica Vilaplana, Laura Pons, Sapir Hochman, Mauricio Eduardo Suárez-Franck, Pedro Luis Fernandez, Julius Drachneris, Donatas Petroska, Renaldas Augulis, Arvydas Laurinavicius, Domingos Oliveira, Diana Montezuma, Anouk B. Bouwmeester, Dominique van Midden, Anne-Marie Vos, Shoko Vos, Jolique van Ipenburg, Maschenka Balkenhol, Koen Winkler, Iris Nagtegaal, Konnie Hebeda, Uta Flucke, Katrien Grünberg, Josef Skopal, Brinder S. Chohan, Jordi Temprana-Salvador, Enrico Munari, Luca Cima, Giulia Querzoli, Yosamin Gonzalez Belisario, Jaeike W. Faber, Geert J. L. H. van Leenders, Jan H. von der Thüsen, Lodewijk A. A. Brosens, Ronald R. de Krijger, Pieter Wesseling, Sandrine Florquin, Mateusz Maniewski, Adam Kowalewski, Robert Barna, Dina Tiniakos, Joan Lop Gros, Rogier Donders, Jake S. F. Maurits, Ming Yang Lu, Chengkuan Chen, Faisal Mahmood, Jeroen van der Laak, Nadieh Khalili, Frédérique Meeuwsen, Francesco Ciompi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, high-stakes cooking competition. On one side, you have three celebrity chefs (AI models) who have read every cookbook in the world. On the other side, you have a panel of 31 human judges, ranging from local home cooks to world-renowned Michelin-starred experts. The challenge? They are all given a single, mysterious ingredient (a microscopic image of tissue) and asked to describe what it is, whether it's "bad" (cancerous), and what specific dish it might be.

This paper, titled DALPHIN, is the report card from that competition. It's the first time someone has organized a fair, open test to see if these "AI chefs" can actually help real doctors (pathologists) diagnose diseases, or if they are just fancy guessers.

Here is the breakdown of what happened, using simple analogies:

1. The Arena: A "Secret Recipe" Contest

Usually, when you test an AI, you give it a practice exam, and then you let it study the answers before the real test. That's cheating.

The DALPHIN team built a secure, locked vault for the answers. They created a dataset of 300 real medical cases from six different countries, covering 130 different types of diseases (from common skin issues to rare tumors).

The Twist: The AI models had to answer questions about these images, but they couldn't see the "correct" answers until after they finished. The answers were kept in a digital vault, only accessible by a secure computer system that graded them automatically. This ensures no one could cheat by memorizing the test.

2. The Contestants

The paper tested three specific "chefs":

GPT-5: A general-purpose AI (like a super-smart encyclopedia that knows a little bit about everything).
Gemini 2.5 Pro: Another general-purpose AI (also very smart, but with a different brain).
PathChat+: A specialist AI trained specifically on medical textbooks and pathology images (like a chef who only cooks Italian food).

3. The Challenges (The Questions)

The AI and the human judges were asked to do four main tasks, mimicking how a doctor looks at a slide:

The "What is this?" Test: Looking at a blurry overview and a zoomed-in detail, can you tell if it's lung tissue, skin, or liver?
The "Is it bad?" Test: Is there a tumor here? (Yes/No).
The "How bad is it?" Test: If there is a tumor, is it harmless (benign), dangerous (malignant), or somewhere in between?
The "What exactly is it?" Test: Give a specific diagnosis name or answer a tricky multiple-choice question about the disease.

4. The Results: Who Won?

The Specialist vs. The Generalists
The specialist AI (PathChat+) was the clear winner in the "diagnosis" category. It performed almost as well as the expert human doctors. It was like a master chef who could taste a sauce and immediately say, "That's a classic Bolognese."

The Generalists (GPT and Gemini) struggled more with specific medical names. They were like general cooks who could tell you it's "soup," but couldn't quite name the specific recipe.

The "Over-Confident" and "Under-Confident" Chefs
The paper found some funny personality quirks in the AI:

Gemini was the "alarmist." It tended to see tumors where there weren't any. It was like a security guard who thinks every shadow is a thief. It was very good at catching real tumors but also cried "wolf" too often.
GPT was the "skeptic." It tended to miss tumors, thinking everything was fine. It was like a guard who ignores a suspicious noise because "it's probably just the wind." It was very good at confirming things were safe but missed the bad stuff.

The Human Element
The human judges were split into three groups:

Experts: The top-tier specialists.
Semi-Experts: Doctors who know the area but aren't the top 1%.
Non-Experts: Residents (trainees) or doctors who don't specialize in that specific body part.

The Big Surprise:
In many tasks, the AI models (especially PathChat+) performed just as well as the non-expert humans and sometimes even the semi-experts. However, in the hardest tasks (like diagnosing rare cancers or complex details), the expert humans still beat everyone, including the AI.

5. The "Context" Trap

The researchers tested two ways of asking questions:

Independent: Asking one question at a time, like a quiz show.
Contextual: Asking questions in a row, where the AI remembers what it said in the previous question.

They found that remembering the conversation (Contextual) helped the AI stay consistent, but it also had a downside: if the AI made a mistake in the first question, it would often double down on that mistake in the second question. It's like a detective who gets the first clue wrong and then builds their whole theory on that error.

6. The Bottom Line

This paper doesn't say "AI is ready to replace doctors." Instead, it says:

AI is getting really good at specific medical tasks, sometimes matching the skill of a trained resident or semi-specialist.
Different AIs have different personalities. Some are too scared (miss things), some are too paranoid (see things that aren't there).
We need a fair way to test them. The DALPHIN benchmark is like a permanent, secure testing ground that will let us track how these AI models improve (or fail) over time without them cheating.

In short, these AI copilots are like very smart, eager interns. They are helpful and often right, but they still need a seasoned expert (a human pathologist) to double-check their work, especially when the case is rare or tricky.

Technical Summary: DALPHIN Benchmarking Digital Pathology AI Copilots

Problem Statement

The emergence of foundation models with visual question answering (VQA) capabilities presents a transformative opportunity for digital pathology, potentially assisting pathologists in routine diagnostics. However, the rapid development of these vision-language models (VLMs) necessitates independent, robust benchmarking to assess their clinical utility and limitations. Existing benchmarks face significant hurdles:

Data Leakage: Publicly available datasets (e.g., PathVQA) risk test data leaking into model training sets, inflating performance metrics.
Lack of Reproducibility: Some benchmarks (e.g., PathQABench) rely on private subsets that prevent the reproduction of results or the benchmarking of new models.
Limited Scope: Many studies lack comparisons across different levels of pathologist expertise or fail to cover the diversity of pathology subspecialties, institutions, and staining protocols.
Clinical Context: Diagnostic interpretation in pathology is subject to interobserver variability, yet few benchmarks contextualize model performance against human experts of varying skill levels.

There is a critical need for a multicentric, open benchmark that sequesters ground truth to prevent leakage while enabling the evaluation of both local and online AI copilots against a diverse human baseline.

Methodology

The DALPHIN Benchmark

The authors introduce DALPHIN, the first multicentric open benchmark for digital pathology AI copilots.

Dataset Composition: The benchmark comprises 300 cases (1,236 histology images) collected from six international healthcare institutions across six countries. It spans 130 diagnoses across 14 pathology subspecialties (including genitourinary, gastrointestinal, soft tissue, breast, thoracic, nephropathology, etc.), covering neoplastic, non-neoplastic, and rare entities.
Image Format: To mimic the pathologist workflow, cases include low-resolution whole-slide image (WSI) overviews and higher-resolution regions of interest (ROIs) extracted from Hematoxylin and Eosin (H&E) or Periodic Acid-Schiff (PAS) stained slides.
Question Design: Each case includes up to six questions:
1. Case-orienting questions: Tissue identification ( $Q_{tissue}$ ), neoplastic status ( $Q_{neoplasm}$ ), and neoplastic behavior ( $Q_{behavior}$ ).
2. Advanced questions: A free-response diagnosis ( $Q_{diagnosis}$ ), a case-specific multiple-choice question ( $Q_{mc}$ ), and a case-specific free-response question ( $Q_{open}$ ).
Ground Truth: Reference answers are provided by the contributing pathologist who had access to the full WSI, clinical context, and ancillary tests. These answers are sequestered and only accessible indirectly via a secure evaluation platform (Grand Challenge) to prevent leakage.

Evaluation Protocol

Models Evaluated: The study evaluated two general-purpose VLMs (GPT-5, Gemini 2.5 Pro) and one pathology-specific VLM (PathChat+).
Generation Scenarios: Models were tested under two scenarios:
1. Contextual (Sequential): Models answered questions sequentially per case, retaining prior context (simulating a diagnostic workflow).
2. Independent: Each question was treated as a standalone query without prior context.
Human Baseline (Reader Study): A reader study was conducted with 31 pathologists (24 board-certified, 7 residents) from 10 countries. Participants were stratified into:
- Subspecialty Experts: >10 years experience in the specific subspecialty.
- Semi-Experts: Routine practitioners in the subspecialty but <10 years experience.
- Non-Experts: Residents or pathologists not routinely practicing in the specific subspecialty.
- Note: Due to resource constraints, not all 300 cases were reviewed by all expertise levels. The dataset was analyzed in nested subsets: $DALPHIN_{full}$ (300 cases, model only), $DALPHIN_{reader}$ (115 cases, 1 expert + 3 non-experts per case), and $DALPHIN_{semi}$ (60 cases, adding 2 semi-experts).

Metrics

Classification Tasks ( $Q_{tissue}, Q_{neoplasm}, Q_{behavior}, Q_{mc}$ ): Evaluated using accuracy, F1 score, Matthews Correlation Coefficient (MCC), precision, recall, specificity, and Cohen's $\kappa$ .
Free-Response Tasks ( $Q_{diagnosis}, Q_{open}$ ): Evaluated using traditional NLP metrics (BLEU-1, ROUGE-L, METEOR, CIDEr) and semantic similarity metrics (BERTScore, BioBERT similarity).
Statistical Analysis: Differences were tested using two-sided paired permutation tests ( $n=1,000$ ) with bootstrapping for confidence intervals.

Key Results

1. Organ Recognition ( $Q_{tissue}$ )

Model Performance: PathChat and Gemini significantly outperformed GPT-5 in organ recognition. PathChat achieved the highest scores, followed by Gemini.
Human Comparison: In the reader subset, expert pathologists significantly outperformed all three VLMs. PathChat and Gemini performed comparably to non-expert pathologists, while GPT-5 scored significantly below all human expertise levels.
Subspecialty Variation: Performance varied by domain; skin was well-recognized by all models, while soft tissue was the most challenging. PathChat frequently omitted tissue predictions for soft tissue cases, indicating "absence of residual normal tissue."

2. Neoplastic Status and Behavior ( $Q_{neoplasm}, Q_{behavior}$ )

Neoplastic Status: PathChat outperformed both general-purpose models in overall classification (F1 and MCC). PathChat's performance was statistically indistinguishable from both experts and non-experts in the reader study.
Bias Analysis:
- Gemini exhibited a sensitivity-oriented bias, tending to overcall neoplasms (high recall, lower specificity).
- GPT-5 exhibited a conservative bias, tending to undercall neoplasms (high specificity, lower recall).
- PathChat showed balanced performance, approaching expert-level recall with precision comparable to non-experts.
Behavior Classification: PathChat again outperformed general models. Confusion matrices revealed that VLMs were more likely to confuse benign and malignant lesions with each other, whereas human pathologists more frequently misclassified them as "in situ" or "uncertain."

3. Diagnostic Reasoning ( $Q_{diagnosis}$ )

Performance: PathChat significantly outperformed GPT-5 and Gemini across all NLP and semantic similarity metrics, approaching the performance of expert pathologists.
Generalization: PathChat's superiority over general-purpose models was consistent across diagnostic difficulty levels (basic, broad, in-depth) and cancer incidence (common vs. rare).
Human Comparison: In the reader subset, experts slightly outperformed PathChat on broad and in-depth diagnoses and rare cancers, while general-purpose models performed below non-expert levels for basic diagnoses.

4. Case-Specific Questions ( $Q_{mc}, Q_{open}$ )

Multiple Choice ( $Q_{mc}$ ): All VLMs showed similar accuracy (74–78%), with no significant differences between them. Experts outperformed all models.
Free-Response ( $Q_{open}$ ): Gemini showed a performance advantage over PathChat and GPT-5 in the full dataset, particularly in clinical categories and specific subspecialties (genitourinary, breast, thoracic). However, in the reader subset, differences between models and human expertise levels were minimal.

5. Contextual vs. Independent Generation

Trade-offs: Most metrics showed no significant difference between contextual and independent scenarios.
Error Propagation: PathChat performed significantly better in the contextual scenario for neoplastic status and behavior tasks. Conversely, general-purpose models achieved higher accuracy on multiple-choice questions in the independent scenario, suggesting that prior context may sometimes introduce anchoring effects or error propagation for these models.

Significance and Claims

The paper claims that DALPHIN establishes a new standard for evaluating digital pathology AI copilots by addressing the critical issues of data leakage and lack of reproducibility.

Robust Benchmarking: By sequestering ground truth and using an indirect evaluation platform, DALPHIN enables the continuous, long-term assessment of rapidly evolving models without compromising dataset integrity.
Clinical Relevance: The inclusion of a diverse human baseline (experts, semi-experts, non-experts) provides a clinically interpretable context for model performance, highlighting that while pathology-specific models (PathChat) can approach expert-level performance in specific tasks, significant gaps remain, particularly in rare cancers and complex reasoning.
Model Behavior: The study reveals distinct behavioral patterns among models (e.g., Gemini's sensitivity vs. GPT's conservatism), which are crucial for understanding potential risks in clinical deployment.
Open Science: The release of images, questions, and evaluation code fosters transparency and allows the community to benchmark future models against a consistent, multicentric standard.

The authors conclude that while AI copilots show promise, their performance is highly task-dependent, and no single metric fully captures their clinical utility. The benchmark serves as a tool to drive the development of more reliable, clinically grounded AI assistants.