Technical Summary: MorphoHELM

Problem Statement

Microscopy-based morphological profiling, particularly the Cell Painting assay, is a cornerstone of drug discovery and functional genomics, capturing holistic cellular responses to perturbations. While deep learning has proliferated as a method for extracting image representations, the evaluation of these representations remains fragmented. Existing studies typically assess models on custom tasks, using disparate preprocessing pipelines and metrics, which prevents fair cross-study comparison. Furthermore, prior benchmarks often focus on narrow settings, failing to systematically quantify how representation quality degrades under varying degrees of technical noise (batch effects). This lack of standardization obscures trade-offs between methods, such as whether a model excels at detecting chemical perturbations but fails at genetic ones, or performs well within a single laboratory but fails across institutions.

Methodology

The authors introduce MorphoHELM (Holistic Evaluation of Learned Morphological Representations), a comprehensive open benchmark designed to standardize the evaluation of feature extraction methods for Cell Painting data.

Benchmark Design

MorphoHELM evaluates representations across three primary biological signal retrieval tasks:

1. Mechanism of Action (MoA) Enrichment: Retrieving compounds with similar mechanisms of action given a query compound.
2. Gene Pathway Enrichment: Retrieving genes with similar functions (e.g., same pathway or protein complex) given a query gene knockout.
3. Replicate Retrieval Consistency: Retrieving biological replicates of the same perturbation.

A core innovation is the evaluation of these tasks across four levels of batch effect stringency to quantify robustness:

• No Restriction (NR): Baseline; candidates can be any sample.
• Not Same Batch (NSB): Candidates from the same experimental batch are excluded.
• Not Same Source (NSS): Candidates from the same institution are excluded (testing domain shifts like hardware differences).
• Not Same Layout (NSL): Candidates sharing the same plate position are excluded (testing position-induced batch effects).

Metrics

The benchmark harmonizes and corrects existing metrics to ensure robustness:

• Enrichment Tasks: The authors report both the Fraction Significant (percentage of queries with statistically significant enrichment via permutation testing) and the Geometric Mean Odds Ratio (OR). They replace the arithmetic mean OR with the geometric mean and apply a modified Haldane-Anscombe correction to prevent infinite values from skewing results when biological classes have few members.
• Replicate Retrieval: The authors utilize k-Nearest Neighbor (kNN) Recall@1 and Mean Average Precision (mAP). They explicitly avoid clustering metrics (like Leiden ARI/NMI) for large-scale screens, demonstrating that such metrics are brittle and driven by latent dimensionality rather than biological signal when the number of unique perturbations is high relative to replicates.

Data and Models

The benchmark aggregates data from the Broad Bioimage Benchmark Collection (BBBC) and the JUMP-Cell Painting (JUMP-CP) consortium. It evaluates a diverse set of representation strategies:

• Classical: Hand-crafted features from CellProfiler.
• Transfer Learning: Natural image foundation models (ImageNet-trained ResNet101, DINOv2).
• Domain-Specific: Microscopy foundation models pre-trained on Cell Painting (OpenPhenom), distinct microscopy datasets (SubCell), and multimodal chemical-structure data (CLOOME).

Preprocessing

To ensure fairness, a standardized postprocessing pipeline is applied to all methods: plate-wise scaling, Principal Component Analysis (PCA) to reduce dimensionality and denoise, and Median Absolute Deviation (MAD) robust standardization. The authors note that this pipeline disproportionately benefits high-dimensional, redundant hand-crafted features (CellProfiler), correcting previous underestimations of their performance.

Key Results

• No Universal Winner: No existing model outperforms classic computer vision analytic strategies (CellProfiler) across all settings and tasks. CellProfiler features remain the strongest general-use representations, particularly for replicate retrieval consistency.
• Foundation Model Performance: Natural image foundation models (specifically DINOv2) generally outperform microscopy-specific foundation models in MoA enrichment tasks, likely due to their expansive self-supervised training data. However, microscopy-specific models do not consistently surpass natural image models.
• Task-Specific Trade-offs: Models exhibit distinct trade-offs. For instance, CLOOME excels at gene pathway enrichment but underperforms in MoA enrichment and replicate retrieval. Conversely, natural image models may capture a broad range of MoAs but miss specific associations found by other models.
• Batch Effect Sensitivity: Performance degrades significantly as batch effect stringency increases. In the "Not Same Source" setting, all models perform below the statistical threshold for significance in MoA enrichment, indicating that retrieving biologically similar compounds across different institutions remains an unsolved problem.
• Replicate Retrieval: In replicate retrieval tasks, CellProfiler features are superior to representation learning strategies across most settings. The authors attribute this to the robustness of hand-crafted features when combined with dimensionality reduction, whereas deep learning embeddings may be more sensitive to individual sample noise without extensive averaging.

Significance and Claims

The paper claims that MorphoHELM provides the first comprehensive, standardized framework to evaluate morphological representations, revealing critical trade-offs that were previously hidden by fragmented evaluation standards.

• Correcting Misconceptions: By standardizing postprocessing (specifically PCA and scaling), the benchmark demonstrates that CellProfiler features have been underestimated in prior literature.
• Robustness over Peak Performance: The benchmark highlights that while some models achieve high scores in lenient settings, they often fail under stringent cross-laboratory conditions. This suggests that current deep learning approaches have not yet solved the problem of detecting biological signal in high-noise, cross-domain environments.
• Methodological Rigor: The authors argue that the choice of metrics and preprocessing significantly alters benchmark conclusions. They specifically caution against using clustering metrics for large-scale screens and advocate for geometric mean ORs to handle sparse biological classes.

Ultimately, MorphoHELM serves as a tool to guide future model development, emphasizing that the goal is not just higher leaderboard scores in ideal conditions, but robust, generalizable representations capable of functioning across diverse experimental realities. The authors make their datasets, code, and evaluation tools publicly available to facilitate this progress.