🧬 biology

Do Pathology Foundation Models Learn Biology? Uncertainty-Based Evaluation on AML Cytomorphology

This study demonstrates that while general-purpose vision models may outperform pathology-specific foundation models on standard clustering metrics, the latter's ability to capture biologically meaningful structure is better revealed through stable uncertainty patterns that distinguish between cell maturity states in acute myeloid leukemia.

Original authors: Aadil Bhat, Sajad Kawa, Mohammad Munzir

Published 2026-08-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aadil Bhat, Sajad Kawa, Mohammad Munzir

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the world of medicine, the microscope has long been the primary tool for understanding the invisible machinery of the human body. When a doctor suspects a blood disorder, they look at a slide of cells, searching for subtle differences in shape, color, and texture that reveal whether a cell is healthy or sick. This task is demanding and prone to human error; even experts can disagree on what they are seeing, and the process is slow. To help, scientists have turned to artificial intelligence, teaching computers to recognize these patterns. Recently, a new generation of powerful computer programs, known as foundation models, has emerged. These are like vast libraries of visual knowledge, trained on millions of images to understand the world. Some are trained on general photographs of nature, while others are trained specifically on medical slides. The big question for doctors and researchers is whether these general-purpose programs, which are incredibly smart about the world at large, can truly understand the specific, complex biology of human cells, or if they need to be trained on medical images to do so.

A team of researchers set out to answer this by testing three different computer models on a specific type of blood cancer called acute myeloid leukemia. This disease occurs when blood cells fail to mature properly, getting stuck in an immature state. The researchers used a dataset containing nearly 17,500 images of individual cells, which had already been sorted by experts into fifteen distinct categories based on their biological development. The goal was to see if the computer models could group these cells together correctly without being told the answers beforehand. They tested a standard model trained on general images, a large model trained on general images but using a more advanced learning method, and a specialized model trained on millions of medical pathology slides.

The results revealed a surprising twist. When the researchers looked at how neatly the computer groups were formed, the general-purpose model trained on natural photographs performed the best. It created tight, well-separated clusters that looked perfect on paper. However, when the researchers checked if these groups actually matched the real biological types of cells, this same model failed completely. It grouped together cells that looked similar on the surface but belonged to entirely different families in the body. In contrast, the model trained specifically on medical slides produced groups that were messier and less geometrically perfect, yet these groups aligned much better with the actual biology of the disease. The specialized model understood the subtle differences that matter to a doctor, while the general model was fooled by superficial similarities.

To get to the heart of why this happened, the researchers developed a new way to test the models: they measured how uncertain the computer was about each cell. In biology, some cells are clearly defined and stable, while others are in a transitional state, changing from one stage to another. These transitional cells are naturally ambiguous. The researchers found that the medical-trained model showed high uncertainty exactly where it should: when looking at these transitional, changing cells. It recognized that these cells were hard to pin down. The general-purpose model, however, showed no such pattern; it was equally confident about everything, failing to sense the biological complexity. This uncertainty turned out to be a more reliable sign of true understanding than the neatness of the groups.

The study also highlighted a critical lesson for how we evaluate these tools. If a researcher only looks at standard scores that measure how tidy the groups are, they would choose the wrong model for the job. The model that looks the best on a standard test is actually the worst at understanding the biology. Furthermore, the researchers found that the performance of the medical model could vary depending on how the experiment was started, but its ability to sense uncertainty remained steady and reliable. This suggests that looking at how a model reacts to ambiguity is a better way to judge its intelligence than simply checking if it sorts things into neat piles. Ultimately, the research shows that training a computer on millions of medical images leaves a distinct mark on how it sees the world, allowing it to grasp the biological reality of disease in a way that general knowledge alone cannot achieve.

Technical Summary: Uncertainty-Based Evaluation of Pathology Foundation Models on AML Cytomorphology

Problem Statement
Pathology foundation models are increasingly utilized as feature extractors for medical image analysis. However, their specific advantage over general-purpose vision models in unsupervised settings remains unverified. While annotation-free methods can group cells by visual similarity, there is no guarantee that these groupings correspond to biologically coherent structures without labeled training data. The core challenge is determining whether domain-specific pretraining (on pathology whole-slide images) yields representations that better capture underlying biological differentiation hierarchies compared to general-purpose self-supervised models or supervised baselines, particularly when ground-truth labels are unavailable at inference time.

Methodology
The study utilizes the AML-Cytomorphology LMU dataset, comprising 17,447 single-cell images spanning 15 expert-defined morphological categories of Acute Myeloid Leukemia (AML). To isolate the effect of the feature extractor, the authors fixed the clustering pipeline (FuzzyGNN-Clust) and varied only the backbone across three distinct pretraining regimes:

ResNet50: Supervised pretraining on ImageNet.
DINOv2-large: Self-supervised pretraining on natural images (142 million photographs).
Virchow2: Self-supervised pretraining on 3.1 million pathology whole-slide images (WSIs).

The evaluation employed a dual-metric approach:

Standard Clustering Metrics: Internal metrics (Silhouette, Davies-Bouldin Index, Calinski-Harabasz Score) and external metrics (Adjusted Rand Index [ARI], Normalized Mutual Information [NMI]) computed post-hoc against the 15 expert labels.
Uncertainty-Based Probing: Per-cell uncertainty scores were calculated to serve as a probe for biological structure. The hypothesis was that uncertainty would reflect known myeloid differentiation biology, specifically showing higher uncertainty for transitional cell types (e.g., myeloblasts, promyelocytes) compared to stable mature types, even in the absence of labels.

Experiments were conducted across five independent random seeds to assess reproducibility.

Key Results

Metric Dissociation (Rank Reversal): A complete inversion was observed between internal and external evaluation rankings.
- Internal Metrics: DINOv2 ranked first (Silhouette = 0.522), followed by ResNet50, with Virchow2 ranking last. DINOv2 produced the most geometrically compact and well-separated clusters.
- External Metrics (Biological Recovery): Virchow2 ranked first (ARI = 0.113), followed by ResNet50, with DINOv2 ranking last (ARI = 0.033). Virchow2 achieved a 2.8× improvement over ResNet50 and 3.4× over DINOv2 in ARI.
Cluster Composition: DINOv2 grouped cells based on compact nuclear morphology (e.g., clustering lymphocytes, segmented neutrophils, and eosinophils together), effectively cutting across lineage boundaries. In contrast, Virchow2 showed partial lineage coherence, grouping monocytic and myeloid precursors (MOB, MYO, MON) together.
Uncertainty Structure: Virchow2 was the only backbone where uncertainty scores significantly differentiated cell types (Kruskal-Wallis $H = 178.2, p = 1.5 \times 10^{-30}$ ). Transitional myeloid precursors exhibited significantly higher uncertainty ( $\bar{\Omega} = 0.384$ ) than stable mature types ( $\bar{\Omega} = 0.313$ ). ResNet50 and DINOv2 displayed flat uncertainty profiles across all 15 cell types.
Reproducibility: While Virchow2's clustering accuracy (ARI) fluctuated widely across seeds ( $0.016 \pm 0.006$ ), its uncertainty structure remained stable (Kruskal-Wallis $H$ ranging 167–178), identifying uncertainty as a more reliable signal of representation quality than clustering accuracy in this context.

Significance and Claims
The paper claims three primary findings:

Biological Alignment of Uncertainty: Virchow2's uncertainty distribution tracks myeloid differentiation boundaries in a manner that neither DINOv2 nor ResNet50 replicates. This pattern persists across five independent runs, suggesting that domain-specific pretraining instills sensitivity to cytological features that natural-image training misses.
The Metric Inversion Problem: The study demonstrates that the backbone producing the most geometrically compact clusters (DINOv2) recovers the least biological structure, while the backbone with the loosest internal geometry (Virchow2) recovers the most. Consequently, benchmarking backbone representations using only label-free internal metrics is unreliable for tasks requiring biological coherence.
Stability of Uncertainty vs. Clustering Accuracy: Virchow2 clustering accuracy is highly seed-dependent, whereas its uncertainty structure is stable. This suggests that in unsupervised scenarios, uncertainty structure is a more robust indicator of representation quality than single-run clustering accuracy.

The authors conclude that domain-specific pretraining leaves a detectable imprint on how a model distributes uncertainty across biologically distinct cell types. This imprint is more informative than standard clustering scores when the objective is biological coherence rather than geometric tidiness. The study treats the link between pathology pretraining and structured uncertainty as a hypothesis requiring further targeted probing, noting limitations such as the use of a stratified subsample (1,577 images) and the testing of only three backbones.

Technical Summary: Uncertainty-Based Evaluation of Pathology Foundation Models on AML Cytomorphology

More like this