Do Geometric Outliers Identify Important Genes in Single-Cell Foundation Models?

Technical Summary: Do Geometric Outliers Identify Important Genes in Single-Cell Foundation Models?

Problem Statement
Single-cell foundation models (scFMs) generate gene embeddings that are increasingly used for biological inference and gene prioritization. However, it remains unclear whether different scFM architectures organize gene space consistently or whether geometric extremes (outliers) within these embeddings reliably identify biologically important genes. A critical gap exists in distinguishing whether a gene's "exceptional" position in an embedding space reflects intrinsic biological properties or is merely an artifact of the specific model's architecture, training objective, and expression encoding. Previous evaluations have focused on downstream task performance, often overlooking whether geometric anomalies are consistent across models or carry genuine functional significance.

Methodology
The study employs a "weight-only" geometric screening approach applied to three distinct scFMs: Geneformer (BERT-style encoder over rank-ordered tokens), scGPT (generative transformer with gene tokens and expression values), and scFoundation (asymmetric encoder-decoder with learned auto-discretization of continuous expression).

1. Geometric Screen: The authors apply a four-metric screen to the static gene-embedding matrices of each model without requiring a forward pass or shared expression data. The metrics are:
   • Embedding norm.
   • Euclidean distance to the vocabulary centroid.
   • Cosine similarity to the centroid.
   • Mean distance to the $k=10$ nearest neighbors (isolation).
     Genes are flagged as outliers if they exceed $|z| > 3$ on any metric.
2. Robustness and Controls:
   • Caller Stability: Outlier sets are re-derived using robust statistical callers (MAD-based and rank-matched) to assess sensitivity to the definition of "extreme."
   • Non-Transcriptomic Control: The same screen is applied to ESM-2 protein-sequence embeddings to determine if geometric outliers recur in sequence space, distinguishing model-specific artifacts from intrinsic protein properties.
   • Matched-Control Deletion: For Geneformer, the 50 highest-anomaly genes (stable across callers) are "deleted" from input sequences in PBMC3k and Tabula Sapiens datasets. Performance is measured via macro-F1 cell-type annotation against 200 (or 100) control sets matched on expression, breadth, gene length, and gene class.
   • Covariate-Adjusted Association: Outlier status is regressed against ClinVar membership (disease relevance) while adjusting for expression, breadth, length, constraint (pLI/LOEUF), and gene class.

Key Results

• Model-Specific Geometries: The three models produced substantially different outlier profiles. While all showed some ribosomal enrichment, strong mitochondrial enrichment was confined to Geneformer and scGPT, with scFoundation showing none. Gene-level agreement was weak; Geneformer and scGPT shared only 72 outliers (a 34-fold excess over chance but a low Jaccard index of 0.172), while scGPT and scFoundation shared only one gene.
• Caller Sensitivity: Geneformer and scFoundation showed high stability in outlier ranking across different statistical callers. In contrast, scGPT's individual outlier membership was highly sensitive to the calling method, attributed to its bimodal embedding norm distribution.
• Lack of Sequence Recurrence: Most geometric outliers did not recur in ESM-2 protein-sequence space (Jaccard $\approx$ 0.054), indicating that the geometry is principally model-specific rather than an intrinsic property of the encoded proteins. Exceptions included mitochondrial genes, which did recur, likely due to their distinct genetic code and hydrophobic nature.
• No Downstream Leverage: Deleting the 50 highest-anomaly genes from Geneformer did not impair cell-type annotation performance more than deleting matched control genes in either PBMC3k or Tabula Sapiens datasets.
• No Adjusted Disease Association: After adjusting for expression, breadth, length, constraint, and class, no model's outlier set showed a detectable association with ClinVar membership. Notably, scGPT showed a strong unadjusted association (OR 2.95) that vanished after adjustment (OR 1.21), suggesting the initial signal was driven by correlated covariates rather than the geometric score itself.

Significance and Claims
The paper concludes that embedding geometry characterizes model-specific structure rather than providing standalone evidence of downstream leverage or disease relevance. The authors assert that:

1. Geometric extremeness is not a proxy for biological importance: Genes identified as outliers are often central (e.g., ribosomal, mitochondrial, constrained), but this centrality is already visible through standard expression and gene class metrics. The geometric score adds no new information for predicting cell-type annotation loss or disease association once these covariates are controlled.
2. Model auditing is distinct from gene prioritization: While geometric screens are useful for comparing how different models represent their vocabularies and for auditing model-specific biases, they should not be used in isolation to nominate genes for experimental follow-up.
3. The value lies in the audit design: The primary contribution is a cheap, checkpoint-level screening method that requires no forward pass or expression data. This allows for immediate model characterization upon release, provided that any hypotheses generated (e.g., "these genes are important") are subsequently tested via matched or covariate-adjusted controls to verify if the model has learned information beyond standard biological priors.

The study emphasizes that findings in one scFM should not be assumed to transfer to another due to significant architectural and encoding differences, and that gene-class enrichments reported without specifying the annotation scheme (overlapping vs. mutually exclusive) can be misleading.