Technical Summary: CellTok – Tokenizing Single-Cell Transcriptomes as a Native Language for LLMs

Problem Statement

Large Language Models (LLMs) excel at processing diverse information when represented as discrete tokens within a shared sequence space. However, single-cell transcriptomes remain a "foreign modality" to LLMs. Unlike natural language or code, transcriptomic data consists of continuous, high-dimensional molecular profiles rather than discrete linguistic units. Consequently, cells cannot be directly read, composed, predicted, or generated as part of an LLM's native token language. Existing approaches often rely on indirect interfaces, such as converting cellular states into textual summaries or aligning continuous embeddings with text via cross-modal learning. These methods limit the ability of LLMs to perform contextual reasoning over cells, such as analyzing cell populations, inferring intercellular communication, or modeling developmental trajectories, because the cellular information is external to the model's token space.

Methodology: CellTok

The authors propose CellTok, a tokenized single-cell language modeling approach that converts transcriptomic profiles into compact, discrete cellular tokens and integrates them directly into the vocabulary of a pretrained LLM.

1. Tokenization Architecture

CellTok employs an encoder–quantizer–decoder architecture inspired by Vector-Quantized Variational Autoencoders (VQ-VAE):

• Input: Single-cell gene expression profiles are normalized and aligned to a shared panel of 5,000 highly variable genes.
• Encoder: Maps the normalized expression profile into a continuous latent representation, which is partitioned into segments.
• Quantizer: Replaces each latent segment with the nearest vector from a learnable codebook (discrete codes).
• Decoder: Reconstructs the gene expression profile from the quantized latent representation.
• Integration: The learned codebook vectors are treated as unique token IDs and added to the vocabulary of a pretrained LLM (specifically Qwen 2.5 7B). This creates an early-fusion design where cellular tokens and text tokens coexist in the same autoregressive modeling framework.

2. Training Strategy

CellTok is trained using two complementary pretraining objectives to align cellular and linguistic modalities:

• Cell Comprehension: The model conditions on a sequence of cell tokens to generate a natural language description of the cellular state.
• Cell Generation: The model conditions on a textual description to generate the corresponding sequence of cell tokens, which are then decoded back into transcriptomic profiles.
• Fine-Tuning: The pretrained LLM is adapted using Low-Rank Adaptation (LoRA), modifying approximately 2% of the parameters (attention projections, feed-forward layers, and token embeddings). The training data includes 68 million high-quality single-cell profiles from CELLxGENE, paired with textual descriptions generated by GPT-4o.

3. Downstream Tasks

The framework supports various tasks by composing multiple tokenized cells within a shared autoregressive context:

• Cell-Level: Cell type annotation and conditional generation of synthetic cells.
• Population-Level: Cluster annotation, heterogeneous population annotation, and disease-state inference (e.g., distinguishing healthy vs. COVID-19 states without explicit cell-type labels).
• Relational: Inferring cell–cell communication (CCC) pathways between source and target populations, and distinguishing communicating vs. non-communicating spatial neighbors.
• Temporal: Inferring developmental ordering (pseudotime) and generating stage-matched transcriptomic profiles along trajectories.

Key Results

The paper evaluates CellTok across diverse benchmarks, demonstrating that LLMs can effectively read and reason over cellular data when provided with native tokens.

• Cell Identity and Generation: CellTok achieves performance competitive with specialized Large Cellular Models (LCMs) like Geneformer and scGPT, and notably outperforms substantially larger general-purpose LLMs (e.g., GPT-4o, LLaMA 3 70B) prompted for single-cell analysis. In generation tasks, it produces synthetic cells that closely match real profiles, outperforming Cell2Sentence and matching scDiffusion.
• Population-Level Inference:
  • Annotation: CellTok outperforms baselines in annotating homogeneous clusters and, crucially, in annotating heterogeneous populations containing confusable cell types (e.g., CD4+ vs. CD8+ T cells) directly from mixed inputs.
  • Disease State: The model accurately identifies disease-associated states (e.g., COVID-19) across mixed populations without being provided cell-type labels during inference, relying instead on transcriptomic patterns and population context.
  • Semantic Sensitivity: Ablation studies show that replacing biologically meaningful labels (e.g., "COVID-19") with neutral labels (e.g., "Status A") reduces performance, indicating that CellTok leverages the semantic priors embedded in the LLM's language knowledge.
• Cell–Cell Communication (CCC):
  • Non-Spatial: CellTok predicts ligand–receptor signaling pathways between cell populations. It generalizes to unseen donors and relies on paired transcriptomic contexts rather than memorizing cell-type identities.
  • Spatial: In spatial transcriptomics (Visium HD), CellTok distinguishes communicating from non-communicating neighboring cells using only transcriptomic input (no spatial coordinates). It generalizes to unseen tissue regions and performs well on heterotypic interactions, suggesting it learns communication competence rather than just cell-type pair frequencies.
• Developmental Trajectories: CellTok successfully infers the relative temporal ordering of cells across developmental stages (brain organoids) and generalizes to unseen time points. It can also generate transcriptomic profiles for specific developmental stages conditioned on cells from other stages, supporting both interpolation and extrapolation.

Significance and Claims

The authors claim that CellTok establishes a unified interface for modeling cells, populations, and biological knowledge in a shared token space.

• Native Language for Biology: The work demonstrates that single-cell transcriptomes can be transformed from a foreign molecular modality into a native language for LLMs. This allows cellular measurements, multi-cell contexts, textual prompts, and biological knowledge to be modeled within a single autoregressive framework.
• Contextual Reasoning: By representing cells as compositional units within the token space, CellTok enables LLMs to perform contextual reasoning over populations, intercellular relationships, and developmental processes, moving beyond treating cells as isolated endpoints.
• Bidirectional Grounding: The framework creates a direct interface between molecular measurements and natural language, where cellular tokens can condition text and text can condition the generation/interpretation of cellular states.
• Proof of Concept: The authors position CellTok as a proof of concept for cellular tokenization as a shared representation and task interface. They acknowledge limitations, such as potential loss of fine-grained quantitative information due to discretization and the need for further evaluation of knowledge transfer across tasks. However, they argue that this approach provides a route toward LLM-based science where models can interpret molecular measurements in closer connection with biological knowledge and scientific questions.