Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures

Technical Summary: Beyond Classification: Pathology Foundation Models as Detection Encoders for Mitotic Figures

Problem Statement
Pathology Foundation Models (FMs), such as UNI, Virchow, and H-Optimus, have demonstrated strong performance in image-level and patch-level classification tasks by leveraging self-supervised learning on vast, unlabeled histopathology datasets. However, their utility in dense object detection tasks remains unclear. Specifically, it is unknown whether the latent spaces of these FMs—trained primarily on image-level self-supervision—contain sufficiently discriminative and spatially resolved features to serve as backbones for detecting mitotic figures (MFs). MF detection is a critical task in oncology for prognostic stratification, yet it demands fine-grained spatial localization rather than semantic summarization. Previous attempts to integrate FMs into cell-level pipelines (e.g., CellViT++) have shown that strong classification representations do not automatically transfer to detection, often underperforming compared to models trained from scratch. This study investigates whether frozen FM backbones can compete with fully end-to-end trained convolutional baselines for MF detection.

Methodology
The authors conducted a systematic evaluation comparing frozen pathology FMs against a conventional convolutional baseline on the multi-domain MIDOG++ dataset and the out-of-domain (OOD) TUPAC16 dataset.

• Backbones: Six ViT-based FMs were evaluated as strictly frozen feature extractors: UNI, UNI2-h, Virchow, Virchow2, H-Optimus-0, and H-Optimus-1. These were compared against two ResNet-50 baselines: one fully fine-tuned end-to-end and one frozen.
• Detection Heads: Each backbone was paired with three distinct detection architectures representing different paradigms:
  • Single-stage: RetinaNet
  • Two-stage: Faster R-CNN
  • Transformer-based set prediction: Deformable DETR
• Architecture Adaptation: Since ViT backbones output single-scale feature maps while detectors require multi-scale hierarchies, a lightweight Feature Pyramid Neck (based on ViTDet design) was inserted between the frozen backbone and the detection head. This neck was the only learnable component in the frozen-backbone configurations, responsible for adapting single-scale representations to multi-scale dense prediction.
• Training Protocol: All models were trained using the MMDetection framework with identical data augmentation (flips, affine transformations, stain jittering) and optimization settings (AdamW, cosine/step learning rate schedules). Evaluation metrics included micro-averaged F1, precision, recall, and the Area Under the Free-range Operating Characteristic curve (FROC-AUC).

Key Results
The study yielded the following performance characteristics on the MIDOG++ test set and the TUPAC16 OOD set:

1. Competitiveness of Frozen FMs: The best frozen FM configuration (H-Optimus-0 with RetinaNet) achieved an F1 score of 0.7718 on MIDOG++, closely approaching the fully fine-tuned ResNet-50 baseline (F1 0.7917). Notably, the frozen FMs significantly outperformed the frozen ResNet-50 baseline (best F1 0.7498), indicating that the advantage stems from pathology-specific pretraining rather than the mere act of freezing a backbone.
2. Impact of Detection Heads: Performance varied significantly based on the detection head. RetinaNet was the most reliable head, achieving the top F1 scores for the ResNet-50 baseline, H-Optimus-0, and Virchow. Faster R-CNN and Deformable DETR showed more variable results depending on the backbone.
3. Out-of-Domain Robustness: On the TUPAC16 dataset, the performance ordering reversed. The best overall F1 score (0.7349) was achieved by a frozen FM (H-Optimus-0 with RetinaNet), surpassing the best ResNet-50 configuration (0.7193). This suggests that frozen FM features may be slightly more robust to domain shifts than fully fine-tuned convolutional networks trained on the source domain.
4. Backbone Ranking: Among the FMs, the pathology-specific H-Optimus models (0 and 1) generally led the performance, followed closely by the Virchow family. UNI and UNI2-h trailed behind.

Significance and Claims
The paper concludes that the frozen latent spaces of current pathology FMs are sufficiently rich and spatially resolved to support dense mitotic figure detection without any task-specific adaptation of the backbone weights. The authors claim that:

• Spatial Information is Present: Despite being trained solely on image-level self-supervision, these models encode the spatial information necessary for localizing small objects like mitotic figures, recoverable by a lightweight neck and head.
• Robustness: Frozen FMs demonstrate a degree of robustness to domain shifts that exceeds that of fully fine-tuned baselines in this specific detection setting.
• Design Implications: The study highlights that the choice of detection head and neck is critical, often outweighing the differences between specific foundation models.

The authors remain cautious, noting that these results characterize the information present in the frozen representations. They suggest that parameter-efficient fine-tuning (e.g., LoRA) could potentially close the remaining gap to end-to-end trained models, though it remains an open question whether such adaptation would preserve the observed out-of-domain robustness. The work establishes that foundation models are viable backbones for dense pathology detection tasks, moving beyond their established role in classification.