Deep Learning for BioImaging: What Are We Learning?

1. Problem Statement

While deep learning has revolutionized natural image analysis through representation learning (acquiring high-level semantic features), its application to microscopy imaging faces a critical uncertainty: what exactly do current foundation models learn?

The authors identify several key issues:

• Ambiguity of Learned Features: It is unclear if models learn biologically meaningful abstractions (e.g., cell morphology, tissue organization) or if they rely on low-level artifacts, batch effects, or dataset-specific shortcuts.
• Benchmark Limitations: Current evaluation metrics often fail to distinguish between genuine biological generalization and performance driven by confounding structures (e.g., batch effects, acquisition artifacts, or simple statistical correlations).
• Lack of Diagnostic Baselines: Unlike natural images where performance is well-understood, microscopy benchmarks often lack simple, interpretable baselines to calibrate whether a model's success is due to learning or architectural priors.

The paper aims to systematically evaluate whether state-of-the-art (SOTA) microscopy foundation models truly acquire high-level biological representations or if their performance can be matched by simpler, untrained, or structural baselines.

---

2. Methodology

The authors conducted a systematic study across two critical biological scales: cell culture (single-cell level) and tissue imaging (histopathology level). They introduced a novel benchmarking framework centered on diagnostic baselines.

A. Datasets and Benchmarks

• Cell Culture:
  • RxRx3-core: A curated dataset for gene-gene and gene-compound retrieval tasks.
  • JUMP-CP: A large-scale dataset for chemical perturbation retrieval (replicate matching).
• Tissue Imaging:
  • HEST-1k: A spatial transcriptomics dataset pairing H&E stained tissue patches with gene expression profiles.
  • HEST-1k-1NN: A coarsened version of HEST-1k where adjacent spatial transcriptomics spots are binned, creating larger patches to test structural organization.
• Reference: ImageNet-1k was used as a natural image baseline to compare representation learning behaviors.

B. Proposed Baselines

To diagnose what models are learning, the authors introduced three specific baseline families:

1. Pixel-Level Statistics: Simple features (mean, standard deviation, skewness) per channel. These test if low-level intensity correlations alone can solve the task.
2. Untrained Models: Deep networks (ResNets, ViTs) with randomly initialized weights used as fixed feature extractors. This isolates the contribution of architectural inductive biases (e.g., convolutional locality, tokenization) from learned biological content.
3. Disentangled Tissue Structure:
   • Cell Graphs: Binary images representing only the spatial positions of cell nuclei centroids (segmented via CellViT), removing all texture, color, and morphology.
   • Point Clouds: Geometric representations of cell coordinates processed by Graph Neural Networks (GNNs).
   • Purpose: To test if spatial organization alone (without appearance) is sufficient for prediction.

C. Experimental Setup

• Models Evaluated: A wide range of foundation models (OpenPhenom, H-Optimus-1, DINOv2/3, UNI, Virchow, etc.) and standard architectures (ResNet, ViT).
• Evaluation Protocol:
  • Frozen Embeddings: Models were evaluated without fine-tuning to assess representation quality directly.
  • Layer-wise Analysis: Performance was tracked across intermediate layers to see if deeper layers improve biological recall.
  • Metrics: Recall@5% for gene retrieval, Mean Average Precision (mAP) for compound retrieval, and Pearson Correlation Coefficient (PCC) for gene expression prediction.

---

3. Key Contributions

1. Systematic Benchmarking with Simple Baselines: The paper establishes a rigorous protocol using untrained models and structural baselines to calibrate microscopy benchmark scores.
2. Discovery of "Shortcut" Learning: The authors demonstrate that SOTA foundation models often perform comparably to untrained models on specific tasks, suggesting that performance is driven by architectural priors and low-level cues rather than learned biological semantics.
3. Diagnosis of Metric Insufficiency: The study shows that standard benchmark metrics often mask the failure of models to learn high-level features. High scores can be achieved by exploiting dataset structure (e.g., batch effects or simple correlations) rather than biological abstraction.
4. Importance of Spatial Organization: In tissue imaging, the authors show that spatial organization alone (cell positions) can be a strong predictor for certain gene expressions, sometimes outperforming or matching complex image-based foundation models, particularly in specific cancer types (e.g., PRAD, COAD).

---

4. Key Results

A. Cell Culture Level (RxRx3-core & JUMP-CP)

• Untrained ViTs vs. SOTA: Surprisingly, untrained Vision Transformers (ViTs) performed comparably to pretrained foundation models (like OpenPhenom) on gene-gene retrieval tasks.
• Minimal Baselines: A minimal untrained single-convolution baseline (SingleConv) was competitive with the best-performing methods.
• Layer Behavior: Unlike natural images (ImageNet), where performance monotonically improves with depth, biological recall on microscopy tasks often decreases or plateaus in deeper layers for untrained models, and pretrained models show inconsistent gains.
• Correlation: Pretrained and untrained ViTs produced highly correlated gene-gene similarity rankings, indicating they rely on the same underlying relational signals (likely low-level or structural) rather than distinct learned biological concepts.

B. Tissue Level (HEST-1k)

• Structure vs. Appearance: Structure-only models (using cell centroids) achieved competitive performance against Out-of-Domain (OOD) image encoders on specific datasets (PRAD, COAD).
• In-Domain Advantage: In-domain foundation models (trained on histology) still showed clear advantages on other datasets (CCRCC, SKCM), suggesting that morphological cues (texture, shape) provide information beyond spatial organization.
• Cell Count Correlation: The authors found a strong correlation between structure-based model performance and simple cell counting. In some cases, the "structure" signal was effectively a proxy for local cell density.
• Gene-Specific Predictability: For certain genes, spatial coordinates alone were sufficient for prediction, while for others, detailed morphological features were required.

C. General Findings

• Representational Collapse: Models with high variance explained by top PCA components (low effective dimensionality) often showed weak biological recall, whereas higher-dimensional embeddings (ViTs, pixel baselines) aligned with better recall.
• Benchmark Specificity: Performance varied significantly across different compounds and genes, indicating that some tasks are "easy" (solvable by shortcuts) while others require true biological understanding.

---

5. Significance and Future Directions

• Re-evaluating Progress: The paper argues that the field's perception of progress in microscopy AI may be inflated. High benchmark scores do not necessarily equate to the acquisition of biologically meaningful representations.
• Need for Diagnostic Benchmarks: Future evaluations must include simple, interpretable baselines (untrained models, structural proxies) to distinguish between genuine learning and dataset-specific shortcuts.
• Interpretability: The results suggest that current models may not be learning the "right" features. Future work should focus on disentangled representations that separate morphology, texture, and spatial organization to better understand what drives predictions.
• Causal Questions: The strong performance of structure-only models raises questions about the causal role of cell arrangement versus cell count in biological function, suggesting a need for controlled interventions in future research.

Conclusion: The paper serves as a critical "reality check" for the bioimaging community, urging a shift from purely performance-driven benchmarking to diagnostic evaluation that measures what models actually learn, ensuring that AI advancements translate to genuine biological insights.