💻 computer science

Benchmarking Deep Learning and Vision Foundation Models for Atypical vs. Normal Mitosis Classification with Cross-Dataset Evaluation

This paper presents a comprehensive benchmark of deep learning and vision foundation models for classifying atypical versus normal mitosis in breast cancer, demonstrating that transfer learning and fine-tuning techniques effectively address classification challenges across both in-domain and newly introduced out-of-domain datasets.

Original authors: Sweta Banerjee, Viktoria Weiss, Taryn A. Donovan, Rutger H. J. Fick, Thomas Conrad, Jonas Ammeling, Nils Porsche, Robert Klopfleisch, Christopher Kaltenecker, Katharina Breininger, Marc Aubreville, Ch

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sweta Banerjee, Viktoria Weiss, Taryn A. Donovan, Rutger H. J. Fick, Thomas Conrad, Jonas Ammeling, Nils Porsche, Robert Klopfleisch, Christopher Kaltenecker, Katharina Breininger, Marc Aubreville, Christof A. Bertram

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a cell dividing is like a baker kneading dough and splitting it into two perfect, identical loaves. This is a normal mitosis. But sometimes, the baker gets clumsy, the dough tears, or the loaves come out uneven. This is an atypical mitosis. In the world of cancer, spotting these "clumsy bakers" is a huge clue that a tumor might be aggressive and dangerous.

However, finding these clumsy bakers is incredibly hard for human pathologists (the doctors who look at microscope slides). They are rare, they look very similar to the normal ones, and even experts often disagree on what they see. It's like trying to find a specific, slightly crooked needle in a massive haystack, where the hay keeps changing color.

This paper is a race to see which computer program is best at spotting these "clumsy bakers."

The Contenders: Three Different Strategies

The researchers set up a competition with three different types of "detectives" (AI models) to see who could solve this puzzle best:

The "Fresh Start" Detectives (End-to-End Models): These are like hiring a smart intern and teaching them everything from scratch using a specific textbook. They learn the rules of cell division entirely from the data provided.
- The Models: EfficientNet, Vision Transformer (ViT), and Swin Transformer.
The "Expert Librarians" (Foundation Models with Linear Probing): Imagine a librarian who has read millions of books about biology but has never been asked to spot clumsy bakers. You ask them to look at a picture and just draw a simple line to say "yes" or "no." They use their vast general knowledge but don't change their brain; they just add a tiny, simple filter on top.
- The Models: Huge pre-trained models like UNI, Virchow, and H-Optimus.
The "Specialized Apprentices" (Foundation Models with LoRA): This is the same librarian, but instead of just drawing a line, you give them a small, specialized notebook (called LoRA) to write down specific rules for this job. They keep their vast general knowledge but learn just enough new tricks to become experts at spotting clumsy bakers without needing to rewrite their whole brain.

The Test Drive: Three Different Courses

To see who was truly the best, the researchers didn't just test them on the same homework they studied. They gave them three different exams:

The Practice Exam (AMi-Br): This was the main dataset used for training. It contained thousands of human breast cancer cell images.
The "Same Town, Different Neighborhood" Exam (AtNorM-Br): This used images from a different source (TCGA). It was like testing the detectives on the same type of city, but with different street signs and lighting.
The "Foreign Country" Exam (AtNorM-MD): This was the hardest test. It included not just human breast cancer, but also dog tumors and other human cancers. It was like sending the detectives to a completely different country with different languages and customs to see if they could still find the clumsy bakers.

The Results: Who Won?

1. The "Specialized Apprentices" (LoRA) were the clear winners.
When the foundation models (the librarians) were given that small specialized notebook (LoRA), they performed incredibly well. The Virchow2 model with LoRA was the champion, achieving the highest accuracy on the practice exam. It proved that taking a massive, general AI and giving it a tiny, specific tweak is a very powerful strategy.

2. The "Fresh Start" Detectives were strong runners-up.
The models trained from scratch (End-to-End) did very well, especially on the "Foreign Country" exam. The Swin Transformer was the best at this, showing that sometimes, a dedicated team trained specifically for the job can handle new, weird situations better than a generalist.

3. The "Expert Librarians" (Linear Probing) struggled.
When the librarians tried to do the job without the specialized notebook (just using a simple line), they performed poorly. It turns out that their vast general knowledge wasn't enough to spot these specific, subtle errors without some extra training.

The Big Takeaway

The paper concludes that while spotting these abnormal cell divisions is a tough job, AI can do it effectively.

The most important lesson is that how you train the AI matters more than just having a big AI.

Simply asking a giant pre-trained model to guess (Linear Probing) isn't enough.
Giving that giant model a tiny, efficient way to learn the specific task (LoRA) works wonders.
Sometimes, training a model from scratch on the specific data is still a very competitive strategy.

The researchers also released all their data and code, essentially handing the "exam questions" and "answer keys" to the rest of the scientific community so everyone can keep improving these detectives. They noted that while the models are good, they still struggle when the "foreign country" (different tissue types or scanners) is too different, suggesting that we still need more diverse data to make these tools perfect for every situation.

1. Problem Statement

The paper addresses the critical challenge of classifying Atypical Mitotic Figures (AMFs) versus Normal Mitotic Figures (MFs) in histopathology.

Clinical Significance: AMFs represent abnormal cell division and serve as an independent prognostic marker for tumor malignancy and patient survival. Quantifying the ratio of AMFs is essential for assessing tumor aggressiveness.
Challenges:
- Low Prevalence: AMFs are rare compared to normal MFs, leading to severe class imbalance.
- Morphological Subtlety: The visual differences between normal and atypical figures are often subtle, with overlapping features.
- Annotation Variability: There is low inter-rater agreement among pathologists, making ground truth generation difficult.
- Domain Shift: Models trained on specific datasets often fail to generalize to data from different institutions, scanners, or species due to variations in staining and tissue preparation.
Gap: While deep learning has improved MF detection, there is a lack of standardized benchmarks and comprehensive evaluations of Foundation Models (pre-trained on massive datasets) specifically for the binary classification of AMF vs. Normal MF.

2. Methodology

2.1 Datasets

The authors introduce and utilize three distinct datasets to ensure rigorous evaluation:

AMi-Br (Training & In-Domain Test): The largest dataset containing 3,720 MFs from human breast cancer (TUPAC16 and MIDOG21 sources). Annotations are based on a 3-expert majority vote.
- Class Balance: ~22.4% Atypical, ~77.6% Normal.
AtNorM-Br (Out-of-Domain Test): Contains 746 MFs from the TCGA breast cancer cohort.
- Annotation: Single expert (highly experienced).
- Purpose: Tests generalization to a different data source (TCGA) with potential label bias.
AtNorM-MD (Out-of-Domain/Multi-Domain Test): Contains 2,107 MFs from a subset of the MIDOG++ dataset, covering six domains (human and canine tumors, including lung, lymphoma, melanoma, etc.).
- Annotation: 5-expert majority vote.
- Purpose: Tests generalization across species and tissue types (significant domain shift).

2.2 Model Architectures & Strategies

The study benchmarks three distinct learning strategies:

A. End-to-End Fine-Tuned Baselines:
- Three architectures trained from scratch (using ImageNet weights) on the AMi-Br dataset: EfficientNetV2, Vision Transformer (ViT), and Swin Transformer.
- Full fine-tuning of all layers.
B. Foundation Models with Linear Probing:
- Eight state-of-the-art Vision Foundation Models (pre-trained on millions of histopathology tiles using self-supervised learning/DINOv2): UNI, UNI2-h, Virchow, Virchow2, Prov-Gigapath, H-Optimus-0, H-Optimus-1, and H0-mini.
- Strategy: The backbone is frozen; only a lightweight linear classifier is trained on top of the extracted embeddings.
C. Foundation Models with LoRA (Low-Rank Adaptation):
- The same eight foundation models are adapted using LoRA, a Parameter-Efficient Fine-Tuning (PEFT) technique.
- Strategy: Original weights are frozen; small, trainable low-rank matrices are injected into attention layers (Query, Key, Value) and MLP components. This allows for adaptation with minimal computational cost while retaining the model's pre-trained knowledge.

2.3 Experimental Setup

Training: All models trained exclusively on the AMi-Br dataset.
Validation: 5-fold cross-validation with patient-level stratification to prevent data leakage.
Metrics: Balanced Accuracy (primary metric due to class imbalance) and AUROC.
Preprocessing: Image resizing to 224x224, standard augmentations (flip, rotation, color jitter), and model-specific normalization.

3. Key Contributions

New Benchmarks: Introduction of two new held-out datasets (AtNorM-Br and AtNorM-MD) specifically designed to evaluate cross-dataset and cross-domain generalization for AMF classification.
Comprehensive Benchmarking: The first large-scale comparison of end-to-end fine-tuning, linear probing, and LoRA fine-tuning for the specific task of atypical vs. normal mitosis classification.
Code and Data Release: Public availability of all code, data, and trained models via a GitHub repository to facilitate future research.

4. Results

Performance Summary (Balanced Accuracy)

In-Domain (AMi-Br):
- Best Model: Virchow2 (LoRA) achieved the highest balanced accuracy of 0.8135.
- Baselines: The Swin Transformer (end-to-end) performed best among baselines (0.8052), outperforming EfficientNetV2 and ViT.
- Linear Probing: Foundation models with only linear probing performed poorly (0.60–0.65), indicating that frozen features were insufficient for this specific task.
Out-of-Domain (AtNorM-Br):
- Best Model: ViT (End-to-End) led with 0.7788.
- LoRA Performance: Virchow (LoRA) followed closely (0.7696). LoRA models consistently outperformed their linear probing counterparts.
Multi-Domain (AtNorM-MD):
- Best Model: Swin Transformer (End-to-End) achieved 0.7723.
- LoRA Performance: Virchow (LoRA) was the top foundation model (0.7705).
- Generalization Gap: All models showed a performance drop compared to AMi-Br, highlighting the difficulty of domain shifts (different species/tissues).

Statistical Significance

LoRA vs. Linear Probing: LoRA significantly outperformed linear probing across all datasets ( $p \ll 0.001$ ).
Baselines vs. Linear Probing: End-to-end baselines significantly outperformed linear probing ( $p \ll 0.0001$ ).
Baselines vs. LoRA: The difference between full end-to-end baselines and LoRA-fine-tuned foundation models was not statistically significant in most cases (e.g., $p=0.73$ for AMi-Br), suggesting that LoRA allows foundation models to reach the performance level of models trained from scratch, but full fine-tuning still holds a slight edge in some cross-domain scenarios.

5. Significance and Discussion

Effectiveness of Adaptation: The study demonstrates that while foundation models possess generalizable features, fine-tuning is essential for the nuanced task of AMF classification. Linear probing is insufficient, but LoRA provides a highly efficient alternative to full fine-tuning, achieving comparable performance with fewer parameters updated.
Domain Shift Challenges: The results confirm that domain shifts (different scanners, staining protocols, and species) significantly degrade performance. The inclusion of multi-domain data (AtNorM-MD) reveals that models trained solely on human breast cancer struggle to generalize to other tissues without specific adaptation.
Future Directions: The authors note that the current limitation is training only on human breast cancer. Future work should integrate these classifiers into a two-stage detection pipeline and expand training data to include diverse tissue types and species to improve robustness.
Clinical Impact: By automating the quantification of AMF ratios, this technology can support more objective, reproducible, and efficient tumor grading, potentially improving treatment planning and prognostic assessments in clinical settings.

In conclusion, the paper establishes that LoRA-fine-tuned foundation models (specifically Virchow2) and end-to-end fine-tuned Swin Transformers represent the state-of-the-art for AMF classification, significantly outperforming simple linear probing and offering a viable path toward automated, robust histopathological analysis.