Towards a Cytometry Foundation Model: Interpretable Sample-level Predictive Modelling via Pretrained Transformers

This paper introduces the Generalised Pretrained Cytometry Transformer (GPCT), an interpretable foundation model that leverages a novel pretraining regime to learn transferable cellular representations from heterogeneous marker panels, thereby overcoming scalability limitations and enabling high-accuracy, biologically validated sample-level predictions across diverse flow cytometry datasets.

The Gist

Imagine you are trying to identify different types of fruit in a massive, chaotic warehouse.

In the world of biology, Flow Cytometry is like a high-tech scanner that zips through millions of individual cells (the "fruit") one by one. For each cell, it measures how bright certain "tags" (markers) are, telling us what kind of cell it is.

The Problem:
For decades, scientists have struggled with two main headaches:

1. The "Different Flashlights" Problem: Every time a scientist runs an experiment, they might use a slightly different set of tags. One lab uses 8 tags, another uses 10, and they might not even use the same colors. It's like trying to sort fruit where one person uses a red light, another uses a blue light, and the labels keep changing.
2. The "Needle in a Haystack" Problem: Sometimes, scientists only have a tiny pile of data (a few hundred cells) to solve a big mystery, but the old computer programs need thousands of examples to learn. They also struggle to explain why they made a decision, which makes doctors hesitant to trust them.

The Solution: GPCT (The "Universal Fruit Sorter")
The authors of this paper built a new AI called GPCT (Generalised Pretrained Cytometry Transformer). Think of it as a super-smart, adaptable robot that has been trained to understand fruit regardless of which flashlight is being used or how many tags are attached.

Here is how it works, using simple analogies:

1. The "Universal Translator" (UCEM Embedding)

Imagine you have a dictionary that translates every possible fruit description into a single, standard language.

• Old way: If a lab didn't measure "Apple Redness," the computer got confused and stopped working.
• GPCT way: It has a special "Universal Translator" that looks at whatever tags are present. If a tag is missing, it doesn't panic; it just says, "Okay, this tag is missing, but I know what the fruit looks like based on the other tags." It turns every messy, different dataset into a clean, standard format that the computer can understand.

2. The "Library of Experience" (Pretraining)

This is the secret sauce. Before the AI tries to solve a specific medical mystery, it spends time reading millions of books in a giant library (the pretraining phase).

• It doesn't need a teacher telling it "This is a sick cell" or "This is a healthy cell."
• Instead, it plays a game of "Fill in the Blanks." The computer hides some of the tags on a cell and tries to guess what they were based on the other tags.
• By doing this billions of times on huge datasets, the AI learns the fundamental rules of biology. It learns that "If a cell has Tag A and Tag B, it's probably a T-Cell," even if it has never seen that specific combination before.
• The Result: When you finally give it a tiny, difficult dataset (like a new disease study), it doesn't start from scratch. It brings its "library of experience" with it, making it incredibly accurate even with very little data.

3. The "Detective's Magnifying Glass" (Interpretability)

Most AI models are "black boxes"—they give an answer, but you don't know how they got there. GPCT is different.

• When GPCT decides, "This sample is from a male mouse," it doesn't just guess. It highlights exactly which cells made it think that.
• It's like a detective pointing at a suspect and saying, "I know it's him because of his shoes and his hat."
• GPCT points to specific groups of cells (like "NK1-1+ KLRG1+ cells") and says, "These specific cells are the reason I made this prediction." This allows real scientists to double-check the AI's work and trust the results.

Why Does This Matter?

• It breaks down walls: You can now mix data from different labs, different machines, and different years, even if they used different equipment.
• It saves time: It automates the tedious work of "gating" (manually drawing circles around cell groups on a screen), which used to take experts hours.
• It works with less data: Because it learned from a "foundation" of massive data, it can solve new problems with very small datasets, which is crucial for rare diseases.

In a nutshell:
The authors built a "Foundation Model" for cell biology. Just as large language models (like the one you are talking to right now) learned to understand human language by reading the whole internet, GPCT learned to understand cells by reading millions of cell scans. It is now ready to help doctors and scientists diagnose diseases and discover new cell types faster and more accurately than ever before.

Technical Summary

1. Problem Statement

Flow cytometry generates high-dimensional single-cell data essential for understanding cellular phenotypes. However, the field faces three critical bottlenecks that hinder the scalability and generalization of machine learning (ML) models:

• Marker Panel Heterogeneity: Different experiments, labs, and time periods use inconsistent sets of markers (antibodies/fluorophores). Traditional ML models require fixed input dimensions, forcing researchers to discard data or train separate models for every panel configuration.
• Data Scarcity: Many clinical or specific biological studies involve small sample sizes (e.g., specific gene knockouts), making it difficult to train deep learning models from scratch without overfitting.
• Lack of Interpretability: While deep learning models (CNNs, Transformers) can predict sample-level labels, they often act as "black boxes," failing to identify which specific cell subsets drive the prediction. This limits biological validation and the refinement of manual gating strategies.

2. Methodology: The Generalised Pretrained Cytometry Transformer (GPCT)

The authors propose GPCT, an end-to-end, interpretable framework designed to learn transferable cellular representations from heterogeneous data. The architecture consists of three core components:

A. Universal Cellular Embedding of Marker Expression (UCEM)

To handle variable marker panels, GPCT introduces a novel embedding strategy that converts any set of measured markers into a fixed-size vector:

• Mechanism: For each cell, the model concatenates a sparse marker expression vector with a one-hot marker availability indicator.
• Handling Missing Data: If a marker is not measured in a specific sample, the model uses a learnable masking value ($\mu_j$) rather than zero, allowing the model to distinguish between "not measured" and "measured but unexpressed."
• Output: This produces a fixed-dimensional token for every cell, regardless of the original panel size, enabling the model to process heterogeneous data natively.

B. Transformer Encoder-Decoder Architecture

• Encoder (Self-Supervised): A modified Transformer encoder processes the UCEM tokens. It uses self-attention to contextualize cellular features, learning complex co-expression patterns and robust representations that are invariant to marker misalignment.
• Decoder (Task-Specific): A cross-attention decoder takes the encoded cell representations and a task-specific learnable query token. It aggregates information to produce a sample-level prediction (e.g., biological sex, gene knockout status).
• Interpretability: The decoder's cross-attention weights provide a quantitative measure of each cell's contribution to the final prediction, enabling cell-level attribution.

C. Two-Stage Training Regime

1. Self-Supervised Pretraining:
   • Objective: The model is trained on vast amounts of unlabeled data using a masked prediction task.
   • Tasks: The model predicts masked marker expression values, percentile ranks, and local density statistics.
   • Masking Strategies:
     • Uniform Random Masking: Masks random cells/markers to learn inter-cell relationships.
     • Marker-wise Masking: Masks entire markers across all cells to learn inter-marker correlations.
   • Goal: To learn a shared latent space of robust cellular patterns that generalizes across different marker panels and batch effects.
2. Downstream Fine-tuning:
   • The UCEM embedding and Encoder are frozen to preserve the pre-trained representations.
   • Only the Decoder and prediction head are trained on the specific downstream task (e.g., classification) using labeled data.

3. Key Contributions

• First Cytometry Foundation Model: GPCT is presented as a foundational model capable of learning from diverse, unlabeled cytometry data to improve performance on downstream tasks with scarce labels.
• Native Cross-Panel Compatibility: Unlike previous methods requiring fixed inputs, GPCT natively handles inconsistent marker panels without data imputation or feature engineering.
• Interpretability via Attention: The model provides direct biological insight by identifying specific cell subsets (via attention weights) that are most influential for a prediction, facilitating the validation of learned patterns against biological knowledge.
• Scalability to Small Data: The pretraining regime significantly boosts performance in low-data regimes (few-shot learning), a critical capability for rare disease or specific genetic mutation studies.

4. Results

The authors evaluated GPCT on two independent mouse immunophenotyping datasets:

• Dataset 1 (Longitudinal): ~14,000 samples with varying marker panels (mostly 8-color).
• Dataset 2 (KOMP): ~72 samples representing 5 specific gene knockouts (KO), with very few samples per class.

Key Findings:

• Superior Performance: GPCT with pretraining achieved 87.0% accuracy and 0.938 AUC on biological sex classification (Dataset 1), outperforming non-pretrained variants and "Decoder Only" models.
• Cross-Panel Robustness: In "leave-one-panel-out" experiments, GPCT showed less than an 8% performance drop when tested on unseen marker panels, demonstrating strong generalization.
• Knowledge Transfer:
  • Pretraining on a mixture of Dataset 1 and 2 significantly improved generalization to Dataset 2.
  • In the Gene KO classification (Dataset 2), the model with a generic pre-trained encoder (trained on external data) achieved a macro-average AUC of 0.919, significantly outperforming models trained only on the small target dataset.
• Interpretability: Attention maps successfully identified biologically relevant cell populations (e.g., $NK1-1^+ KLRG1^+$ cells) associated with biological sex, consistent with known immunology.
• Few-Shot Learning: In few-shot scenarios (1–8 samples per class), the pre-trained model (Model D) consistently outperformed the non-pretrained model (Model C), which often performed near random chance.

5. Significance

This work represents a pivotal step toward a cytometry foundation model. By leveraging the Transformer architecture and self-supervised pretraining, GPCT addresses the fundamental challenges of data heterogeneity and scarcity in flow cytometry.

• Clinical Impact: It enables the integration of disparate clinical datasets that were previously incompatible due to different marker panels, paving the way for large-scale, data-driven immune profiling.
• Scientific Discovery: The interpretability features allow researchers to move beyond "black box" predictions, using the model to discover novel cell subsets and refine manual gating strategies.
• Future Outlook: The authors suggest that applying this paradigm to the full corpus of available flow cytometry data could create a comprehensive repository of generic cellular patterns, analogous to Large Language Models (LLMs) in NLP, revolutionizing diagnostic and research workflows in immunology and precision medicine.