💻 computer science

SHIFT: Survival Prediction from Incomplete and Heterogeneous Genomic Data

The paper introduces SHIFT, a missingness-aware transformer model that enables robust survival prediction across heterogeneous genomic datasets by directly processing incomplete feature inputs without requiring imputation or restricting analysis to shared genes.

Original authors: Muhammet Sami Yavuz, Ayhan Can Erdur, Sabri Mustafa Kahya, Benedikt Wiestler, Jana Lipkova

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Muhammet Sami Yavuz, Ayhan Can Erdur, Sabri Mustafa Kahya, Benedikt Wiestler, Jana Lipkova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how long a patient might live based on their genetic "fingerprint." Usually, doctors and scientists have a huge problem: every hospital uses a different genetic testing kit. One hospital might check 200 genes, while another only checks 20. It's like trying to bake a cake using a recipe that lists 200 ingredients, but your local grocery store only sells 20 of them.

For a long time, scientists tried to fix this by either throwing away the patients who didn't have the full 200 ingredients (losing valuable data) or by guessing what the missing ingredients might have been (imputation). But guessing is risky; if you guess the wrong spice, your cake tastes terrible, and your prediction is wrong.

Enter SHIFT, a new AI model that acts like a super-flexible chef. Instead of needing the full list of 200 ingredients to start cooking, SHIFT can look at whatever ingredients are actually in the kitchen and make a great prediction right away. It doesn't guess the missing stuff; it just focuses on what's there.

The Magic Trick: The "Missing" Mask

Think of SHIFT as a detective who wears a special pair of glasses. When the detective looks at a patient's genetic data, if a gene is missing (because the hospital didn't test for it), the glasses simply turn that spot black. The detective ignores the black spots and solves the mystery using only the clues that are visible.

The paper shows that this detective is incredibly smart. In tests with two types of cancer—glioblastoma (a brain cancer) and lung squamous cell carcinoma—SHIFT worked just as well as the best traditional methods when all the data was perfect. But when the data was messy and missing huge chunks (like the lung cancer study where one group was missing 175 out of 197 genes, or 88.8% of the data!), SHIFT didn't stumble.

While other methods tried to "fill in the blanks" using guesswork (like guessing the missing genes based on neighbors), SHIFT just used the real data it had. In the lung cancer test, SHIFT matched the performance of the best guessing method without ever making a single guess.

Training with "Blindfolds"

How did the detective get so good at ignoring missing clues? The authors used a clever training trick called Variable-Rate Masking (VRM).

Imagine you are training a student for a math test. Usually, you give them the full worksheet. But with VRM, the teacher randomly covers up different numbers on the worksheet for every practice problem. Sometimes they cover one number, sometimes half the page. The student learns to solve the problem using whatever numbers are left visible.

The paper suggests that this "blindfold" training makes the model much tougher. Even when the model is tested later with a full worksheet (no missing data), it performs better than models that were trained without the blindfolds. It's like a student who practiced with missing numbers becoming so good at logic that they ace the test even when all the numbers are there.

The "Incomplete" Teammate

Here is another surprising finding: You don't have to kick out the incomplete data.

In the study, the researchers tried adding a group of patients who only had 22 genes tested (out of the usual 197) into the training mix. Normally, scientists would throw this group away because their data is "too incomplete." But the paper suggests that when you use SHIFT, you can actually use this incomplete group to help the model learn.

When they added these 102 patients with limited data to the training, the model's predictions for a completely different group of patients (the CPTAC cohort) got slightly better. The authors suggest that even a "half-empty" dataset can teach the model something useful if the model knows how to handle the missing pieces.

What This Doesn't Mean

It's important to know what this paper doesn't say.

It doesn't say SHIFT is a magic cure for cancer. It only predicts survival risk.
It doesn't say this works for every type of cancer yet. The authors only tested it on brain and lung cancers.
It doesn't say you should stop using other methods entirely. The paper suggests SHIFT is a strong alternative, especially when data is messy, but it admits that more testing on different diseases is needed.
It doesn't claim that guessing (imputation) is always bad. It just shows that when the missing data is huge and structural (like a whole section of the genome never being tested), guessing is less reliable than just using what you have.

The Bottom Line

The paper suggests that we can build a single, smart AI model that works across different hospitals, even if those hospitals use different genetic testing kits. By teaching the model to ignore missing data instead of guessing it, and by training it with "blindfolds" to make it tougher, we can include more patients in our studies and get better predictions. It's a step toward making precision medicine work for everyone, not just the lucky few who happen to have the perfect genetic test kit.

Technical Summary: SHIFT – Survival Prediction from Incomplete and Heterogeneous Genomic Data

1. Problem Statement

Genomic prediction models for patient survival often fail to generalize across medical institutions due to structural feature missingness. Unlike random missing data, genomic missingness is often systematic: different centers utilize different sequencing panels, measuring distinct subsets of genes. This creates a scenario where a model trained on a broad panel (e.g., 197 genes) cannot be directly deployed on a site using a narrow panel (e.g., 22 genes) without significant data loss or imputation.

Existing approaches to this challenge typically rely on one of three suboptimal strategies:

Restricting analysis to the intersection of features shared across all cohorts, discarding potentially informative genomic data.
Excluding patients with incomplete profiles, reducing cohort size and diversity while amplifying selection bias.
Test-time imputation (e.g., KNN, mean imputation), which can distort biological signals, introduce bias, and hallucinate clinically relevant features, particularly when large blocks of features are structurally absent.

The paper argues that these limitations hinder the deployment of robust, multi-center survival models in precision oncology.

2. Methodology: The SHIFT Model

The authors propose SHIFT (Survival prediction Handling Incomplete Features using Transformer), a missingness-aware survival model designed to predict outcomes directly from incomplete genomic inputs without test-time imputation.

Architecture

SHIFT is a transformer-based architecture adapted for tabular genomic data, consisting of three main components:

Feature Embedding Block: Instead of processing the full genomic vector jointly, SHIFT maps each genomic variable independently into a token representation using a Self-Normalizing Network (SNN) block (Linear $\to$ SELU $\to$ AlphaDropout). Each feature $x_l$ is embedded via a dedicated $SNN(1, D)$ where $D=128$ . Missing features are zero-filled as placeholders but are not processed as valid inputs.
Masked Transformer Encoder: A 2-layer Transformer encoder processes the sequence of feature tokens. Crucially, it employs a missingness mask ( $M$ ) as a key-padding mask. This ensures that tokens corresponding to missing features are excluded from the self-attention mechanism. Consequently, the learnable [CLS] token aggregates information only from observed genomic variables. Positional embeddings are omitted as genomic features lack a natural sequential order.
Survival Prediction Block: The final [CLS] representation is passed through an SNN and a linear layer to produce logits for $K=4$ discrete time intervals. A sigmoid activation converts these into discrete hazard probabilities. The model is trained end-to-end using a discrete-time negative log-likelihood loss, which accounts for censored data.

Training Strategy: Variable-Rate Masking (VRM)

To enhance robustness to heterogeneous missingness patterns, the authors introduce Variable-Rate Masking (VRM) during training.

For each training epoch and patient, a random fraction of features ( $k_i$ ) is masked, where $k_i$ is sampled from a uniform distribution up to a maximum masking fraction $f$ .
This forces the model to learn inter-feature relationships from partially observed inputs, simulating the variability seen across different institutions.
The default setting uses $f=0.5$ , meaning up to 50% of features are randomly masked during training.
Crucially, VRM is only used during training. At inference, the model handles naturally missing features natively via the missingness mask, requiring no imputation.

3. Key Contributions

SHIFT Model: Introduction of a transformer-based model capable of survival prediction from incomplete genomic data without imputation, utilizing masked self-attention to focus solely on observed inputs.
Variable-Rate Masking (VRM): A training strategy that improves model robustness to cross-cohort shifts by exposing the model to diverse subsets of observed features during training.
Inclusive Modeling: Demonstration that severely incomplete cohorts (e.g., those with 88% missing features) can be included in model development to improve performance on external data, rather than being discarded.

4. Experimental Results

The authors evaluated SHIFT on two cancer types: Glioblastoma (GBM) and Lung Squamous Cell Carcinoma (LUSC), using public (TCGA, CPTAC) and private cohorts.

Performance on Complete Data

In settings where training and test data share the same feature space (no missingness), SHIFT (with or without VRM) performs competitively against standard baselines (CoxPH, RSF, DeepSurv, SNN).
VRM Impact: While SHIFT without VRM is competitive, adding VRM significantly improves cross-institutional generalization (e.g., from TCGA to CPTAC/German cohorts), suggesting VRM acts as a regularizer against overfitting to specific feature subsets.

Performance on Incomplete Data (LUSC)

Scenario: The US cohort had only 22 of the 197 features available (88.8% structural missingness).
Baselines: Standard models trained on the full 197 features required KNN or mean imputation at test time. These approaches showed high sensitivity to the imputation strategy, with many baselines degrading significantly (e.g., SNN dropping to a C-index of ~0.42 with KNN).
SHIFT: SHIFT-VRM, trained on the full 197 features and evaluated natively on the 22-feature US cohort (no imputation), achieved the best ensemble performance (C-index 0.570), matching the strongest imputed baseline (CoxPH with mean imputation) and outperforming most others.
Control: When all models were restricted to the shared 22 features, SHIFT-VRM did not outperform the best baseline (SNN), confirming that SHIFT's advantage stems from its ability to leverage richer training data while remaining robust to deployment constraints, rather than just the transformer architecture itself.

Multi-Site Training

The authors tested whether incorporating the incomplete US cohort (22 features) into the training set (alongside TCGA) improved performance on the external CPTAC cohort.
Result: Including the incomplete US cohort improved the C-index on CPTAC from 0.602 (TCGA-only) to 0.629 (TCGA+US). This supports the claim that partially observed cohorts provide useful signals and need not be excluded.

Ablation Study

Varying the masking fraction $f$ showed that $f=0.5$ provided the most consistent performance across cohorts.
Removing VRM ( $f=0.0$ ) consistently weakened external generalization, even in settings where test data was complete, confirming VRM's role as a regularizer.

5. Significance and Claims

The paper claims that SHIFT offers a practical strategy for multi-center survival prediction in precision oncology by addressing the "structural missingness" inherent in genomic data heterogeneity.

Single Model Deployment: A single model can be trained on heterogeneous data and deployed across institutions with different sequencing panels without site-specific retraining or imputation.
Robustness to Structural Missingness: Conditioning predictions only on observed inputs (via masking) can be as robust as, or more robust than, reconstructing large blocks of unmeasured features via imputation, particularly when missingness is non-random and structural.
Data Inclusivity: The study challenges the practice of excluding incomplete cohorts. By handling missingness natively, models can leverage data from centers with narrower panels, increasing sample size and diversity without compromising performance.

The authors acknowledge limitations, including the restriction to two cancer types, modest external cohort sizes, and the use of random masking rather than site-structured masking patterns. However, they conclude that explicitly modeling feature availability is a viable path toward more deployable and inclusive genomic survival models.