💻 computer science

CytoFormer: A Molecularly Supervised Cell Foundation Model for Histopathology Cell Classification

CytoFormer is a novel cell foundation model that leverages paired H&E histology and spatial transcriptomics data from 16 organs to achieve accurate, label-efficient, and generalizable cell classification without relying on manual pathologist annotations.

Original authors: Jialu Yao, Songhao Li, Alina Yu, Zhi Huang

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jialu Yao, Songhao Li, Alina Yu, Zhi Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medicine, the most common way to look inside the human body is through a microscope, examining thin slices of tissue stained with pink and purple dyes. This is the standard of care for diagnosing diseases like cancer. A pathologist looks at these slides and identifies the different types of cells present—whether they are healthy tissue, invading immune cells, or rogue tumor cells. This visual inspection is powerful, but it is also slow, subjective, and difficult to scale. Counting millions of individual cells by eye is impossible, and even experts can disagree on what a specific cell is, especially when different cell types look nearly identical under a microscope. For decades, the dream has been to teach computers to do this counting automatically, but building such a computer has been stuck in a bottleneck: it requires a human expert to label millions of cells by hand to teach the machine what to look for. This manual work is tedious, expensive, and often unreliable, leaving the field without enough high-quality data to train truly intelligent systems.

A team of researchers at the University of Pennsylvania has found a way to bypass this human bottleneck entirely. Instead of asking pathologists to label cells, they used a different kind of biological map to teach the computer. They paired the standard pink-and-purple tissue slides with a newer technology called spatial transcriptomics, which acts like a molecular barcode scanner. This scanner reads the genetic activity inside each individual cell while it is still sitting in the tissue. Because the genetic code is unique to a cell's identity, it tells the researchers exactly what type of cell they are looking at, without any human guessing. By matching these molecular identities to the corresponding images on the standard slides, the team created a massive, self-supervised dataset. They used this to train a new artificial intelligence model called CytoFormer, which learned to recognize cell types just by looking at the shape and texture of the cells in the routine microscope images.

The result is a system that can identify cell types with a level of accuracy that rivals human experts, but without needing a human to draw a single box around a cell during training. The researchers assembled data from 81 different tissue sections covering 16 different organs in the human body, ranging from the breast and lung to the brain and bone marrow. In total, they processed more than 15 million individual cells. For each cell, the computer learned to associate the visual appearance in the stained image with the specific molecular identity provided by the genetic scanner. This allowed the model to learn a universal language of cell shapes that works across the entire body. When tested on new tissue sections that the model had never seen before, it correctly identified the cell types in 85 percent of cases on average. More importantly, the model did not just count cells; it reconstructed the entire architecture of the tissue, correctly mapping out where tumors, immune cells, and healthy structures were located, effectively reproducing the biological landscape of the organ.

What makes this discovery particularly significant is how well the model transfers its knowledge to new situations. The researchers tested whether the AI could handle organs and cell types it had never encountered during its training. They applied the model to four different public datasets containing cells from organs like the colon and skin, which were not part of the original training set. Even without seeing these specific tissues before, the model outperformed six other leading artificial intelligence systems that had been trained on millions of manually labeled images. It was especially effective in difficult scenarios where cells look very similar, such as distinguishing between normal healthy tissue and cancerous tissue that mimics it. In one test, the model learned to spot normal cells hidden among look-alike tumor cells using only a few hundred examples provided by a human, whereas other systems required many more examples to reach the same level of accuracy. This suggests that the model learned the fundamental visual rules of how cells look in their environment, rather than just memorizing a list of specific cell types.

The researchers also explored how much of the surrounding tissue the computer needs to see to make a correct identification. They tested different sizes of the image window around each cell, from a tiny view showing just the cell itself to a large view showing the whole neighborhood. They found that the model worked best when it could see the cell and its immediate neighbors, but not the entire tissue section. This balance allowed the computer to understand the context of the cell—whether it was sitting in a blood vessel, a tumor mass, or a healthy gland—without losing the fine details of the cell's own shape. This specific window size, which corresponds to a very small but clear view of the cell, proved to be the sweet spot for accuracy.

By turning the molecular data from the genetic scanner into a teaching tool for visual recognition, the team has created a reusable resource that can be applied to any routine tissue slide. The model does not require expensive new equipment to run; it can analyze standard slides that are already being used in hospitals every day. The researchers have made the code and the trained model available to the scientific community, allowing others to use this new way of seeing cells. While the current model groups some very similar cells together, such as different types of immune cells, because they look alike, it represents a major step forward. It proves that we can build powerful tools for cellular analysis without relying on the slow and expensive process of manual labeling. This approach opens the door to analyzing millions of cells across thousands of patients, turning routine microscope slides into detailed maps of cellular activity that could help doctors diagnose diseases earlier and more accurately.

Technical Summary: CytoFormer

Problem Statement

Routine haematoxylin and eosin (H&E) histology contains critical diagnostic information regarding cell types, their proportions, and spatial arrangements. However, automating cell-level analysis at scale has been hindered by the "labeling bottleneck." Traditional approaches rely on manual annotations by pathologists, which are slow, expensive, and often unreliable for distinguishing morphologically similar cell types (e.g., macrophages vs. fibroblasts vs. poorly differentiated tumor cells). Consequently, existing large-scale datasets contain only on the order of $10^5$ nuclei with coarse categories, and current pathology foundation models are primarily pretrained on whole-slide or patch-level tasks, often overlooking fine-grained cell-level features.

Methodology

Dataset Curation: Paired Spatial Transcriptomics and H&E

To bypass manual annotation, the authors constructed a pan-tissue single-cell H&E dataset by leveraging imaging-based spatial transcriptomics (specifically the Xenium platform).

Data Source: 81 paired Xenium sections spanning 16 human organs (breast, lung, ovary, lymph node, colon, tonsil, cervix, brain, kidney, pancreas, liver, skin, prostate, bone marrow, heart, and bone) were assembled from the 10x Genomics portal and HEST-1k.
Label Generation Pipeline:
1. Clustering: Unsupervised clustering of single-cell expression profiles.
2. Annotation: Marker-gene based annotation of clusters.
3. Human Review: Organ-wise expert review of clusters to ensure biological plausibility.
4. Confidence Filtering: A per-cell confidence gate based on neighbor purity and marker margin was applied to remove ambiguous cells (e.g., doublets or boundary cells).
5. Taxonomy Mapping: Clusters were mapped to 23 cell types commonly reported in each organ. This includes 7 pan-tissue classes (Tumour, Stroma, Epithelium, Lymphocyte, Endothelium, Macrophage, Plasma cell) and 16 organ-specific classes (e.g., Hepatocyte, Cardiomyocyte).
Final Dataset: 15.4 million cells, each with a paired 56 µm H&E image patch and a molecularly derived cell type label.
Split Strategy: To prevent data leakage, training and test sets were split spatially rather than randomly. For organs with multiple sections (breast, lung), whole sections were held out. For others, sections were bisected into spatial blocks with a 112 µm guard band to ensure no test cell shares a field of view with a training cell.

Model Architecture: CytoFormer

CytoFormer is a cell foundation model designed for multi-task, per-organ classification.

Encoder: A ViT-giant transformer initialized from UNI2-h. It processes 224×224 pixel crops (derived from 56 µm fields of view) and outputs a 1536-dimensional embedding per cell.
Heads: Instead of a single global classifier, the model employs 16 linear classifiers (one per organ). The organ identifier selects the appropriate head during inference. This design allows the shared encoder to learn universal morphological features while the heads specialize in the specific cell type distributions of each organ.
Training: The encoder and all heads are trained jointly on the 12.1 million training cells using AdamW with a multi-task objective.

Evaluation Protocols

The model was evaluated under three distinct regimes:

Spatially Held-Out: Direct inference on unseen tissue sections from the same 16 organs.
Linear Fine-Tuning: The encoder is frozen, and a linear head is trained on four public, expert-annotated benchmarks (PanNuke, CoNSeP, MoNuSAC, PUMA) to test representation transferability.
Full Fine-Tuning: End-to-end training of the backbone and head on the same public benchmarks.
Active Learning: An interactive setting on a colorectal cancer VisiumHD section to detect normal epithelium amidst look-alike tumor cells with minimal annotations.

Key Results

Performance on Held-Out Tissue

On the spatially held-out test split (2.9 million cells), CytoFormer achieved:

Accuracy: 0.846
Macro-F1: 0.779
Tissue Architecture: Visual inspection of a held-out breast section showed that predictions reproduced the reference tissue architecture, with errors primarily occurring at the level of individual cells (e.g., misclassifying mononuclear cells) rather than structural patterns.
Ablation: A 56 µm crop size was found to be optimal, balancing the need for nuclear resolution with sufficient tissue context.

Transfer Learning and Benchmarking

CytoFormer demonstrated superior generalization compared to six existing pathology foundation models (UNI2-h, Virchow2, MUSK, CONCH, PathGen, PLIP):

Linear Fine-Tuning: CytoFormer achieved the best macro-F1 in 24 out of 25 organ-specific settings across the four benchmarks. It significantly outperformed baselines on difficult classes (e.g., endothelial cells, connective cells) and on organs/cell types not seen during pretraining (e.g., bladder, thyroid, melanophages).
Full Fine-Tuning: Even when all models were fully fine-tuned, CytoFormer maintained a lead, particularly on challenging classes, confirming that its pretraining on molecularly supervised data provides a robust starting point.

Label Efficiency

In an active learning setting for detecting normal epithelium in colorectal cancer:

CytoFormer reached an F1 of 0.82 with only a few hundred annotations, leading the strongest baseline (UNI2-h) by 0.13 F1.
The learning curve for CytoFormer rose significantly faster, achieving most of its final performance within the first ~200 annotations, whereas baselines plateaued at lower performance levels.

Significance and Claims

The paper claims that CytoFormer demonstrates that paired spatial transcriptomics can replace manual pathologist annotation as a source of cell-level supervision for H&E histology. By leveraging molecular identity to supervise morphology, the authors created a dataset of 15.4 million cells without manual labeling.

The significance of the work lies in:

Scalability: Generating a massive, high-quality cell-level dataset that scales far beyond manual annotation capabilities.
Representation Learning: Proving that a foundation model trained on this molecularly supervised data learns generic, transferable cell-level features that generalize to organs and cell types unseen during training.
Label Efficiency: Providing a representation that enables accurate cell typing with minimal human intervention, making it practical for clinical tasks where annotations are scarce.
Foundation for Downstream Tasks: The model outputs a map of typed cells for any routine H&E slide, providing a cell-level representation that can serve as a feature source for clinical tasks such as diagnosis, prognosis, and treatment response prediction.

The authors note limitations, including that the label space is restricted to commonly reported cell types (not subtypes like T vs. B cells) and that the current work relies solely on Xenium data, though future work may incorporate other modalities like spatial proteomics.