Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views

Technical Summary: Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views

Problem Statement
Current foundation models for single-cell transcriptomics are predominantly pretrained via masked expression-value prediction (Masked Language Modeling, or MLM). While this objective effectively learns gene-level dependencies and latent co-expression structures, it provides supervision only at the individual gene level. Consequently, these models do not directly optimize whole-cell representations, which are critical for downstream tasks such as cell-type annotation and gene regulatory network (GRN) inference. Furthermore, strong reconstruction performance does not guarantee discriminative or transferable cell embeddings.

Additionally, applying standard contrastive learning to single-cell data presents two domain-specific challenges:

1. Positive-Pair Construction: Unlike image augmentations (e.g., cropping) that preserve object identity, perturbing gene expression values can alter the underlying biological state or cell identity, making it difficult to construct valid positive pairs.
2. Negative-Pair Shortcuts: In sparse transcriptomic inputs, different cells often retain different subsets of gene identities. Standard contrastive learning risks allowing models to distinguish negative pairs based solely on gene-set composition (gene identity) rather than learning the correspondence between specific genes and their expression values.

Methodology: CoCoS
The authors propose CoCoS (Co-expression-guided Contrastive pretraining for Single-cell), a framework designed to learn cell representations through complementary transcriptomic views while addressing the challenges above. The method integrates three specific adaptations:

1. Co-expression-Guided Gene Partitioning:
   To construct biologically valid positive pairs without altering cell identity, CoCoS partitions the genes of each cell into two complementary views based on a co-occurrence graph. Genes that frequently co-occur in highly expressed states are assigned to different views. For each cell, the model generates two views: one masks expression values in the first gene set, and the other masks values in the second set. This ensures that every gene value is observed in one view and masked in the other, preserving the original gene identities and expression values while providing complementary information.

2. Expression-Aware Contrast-Set Construction:
   To prevent the model from using gene-set identity as a shortcut, CoCoS constructs "hard negatives." For a given anchor cell view, the model creates identity-matched negatives by permuting the expression values while keeping the gene identities and the masking pattern unchanged. Since the original and permuted inputs contain the exact same set of genes, the model cannot distinguish them based on gene composition alone; it must learn the specific correspondence between genes and their expression values. The contrast set includes these hard negatives alongside standard in-batch negatives from other cells.

3. Competence-Gated Contrastive Onset:
   The framework introduces a data-dependent controller to determine when to activate the contrastive objective. Early in training, the model has not yet learned stable cross-gene dependencies through masked reconstruction. Introducing contrastive loss immediately might cause the model to align views using technical cues (e.g., library size or sparsity) rather than biological correspondence. CoCoS uses a fixed, unlabeled "sentinel" cohort to monitor three metrics: gene-program concordance (separation of paired vs. mismatched views), reconstruction error, and embedding diversity (repertoire). The contrastive weight remains zero until these metrics indicate the model has achieved stable reconstruction competence and cross-view structure, after which the weight ramps up smoothly.

Key Contributions

• Identification of Mismatch: The authors characterize the misalignment between masked expression-value reconstruction and whole-cell representation learning, defining three specific requirements for applying contrastive learning to single-cell pretraining: biologically valid positive views, mitigation of gene-identity shortcuts in negatives, and appropriate timing for contrastive supervision.
• CoCoS Framework: The introduction of a pretraining framework that jointly designs co-expression-guided views, expression-aware contrast sets with fixed-identity value derangements, and a competence-gated onset mechanism.
• Empirical Validation: Extensive experiments demonstrating that bounded whole-cell contrastive supervision yields more discriminative and transferable representations than pure reconstruction-based methods.

Results
The authors evaluated CoCoS (specifically the CoCoS-GPC variant) on downstream tasks using frozen whole-cell embeddings:

• Cell-Type Annotation: Under a unified ten-dataset cell-forward evaluation protocol, CoCoS-GPC achieved the highest mean k-NN accuracy (74.82%) and linear probe accuracy (78.85%) among the compared variants, outperforming established models like scGPT and Geneformer V2-104M.
• Gene Regulatory Network (GRN) Inference: In a six-network GRN evaluation, CoCoS-GPC recorded the highest mean AUROC (0.8733) and AUPRC (0.7120) point estimates. The authors note that while the mean is highest, the best-performing variant differs across individual networks.
• Ablation Studies:
  • View Design: Expression-aware sampling outperformed uniform sampling and balanced-random panels.
  • Negative Construction: Diagnostic tests confirmed that fixed-identity negatives successfully mitigated reliance on gene-set composition, narrowing the confidence gap between ordinary negatives and identity-matched derangements.
  • Onset Timing: A competence-gated onset (triggered at step 93,000) outperformed immediate contrast and fixed warmup schedules (1 or 2 epochs), validating the necessity of waiting for reconstruction competence before introducing contrastive supervision.

Significance and Claims
The paper claims that complementary-view contrastive learning is an effective direction for extending single-cell pretraining beyond gene reconstruction. By addressing the specific challenges of positive-pair validity, negative-pair shortcuts, and training timing, CoCoS establishes a method that produces more discriminative and transferable cell representations. The authors position this work not as a replacement for reconstruction but as a necessary complement that optimizes the geometry of whole-cell embeddings, thereby improving performance on downstream tasks like cell-type annotation and GRN inference within the evaluated datasets and protocols. The results suggest that properly bounded whole-cell contrastive supervision is superior to pure reconstruction-based approaches for learning transferable cell representations.