Dynamic multimodal survival prediction in multiple myeloma integrating gene expression, longitudinal laboratories, and treatment history

This paper presents a dynamic multimodal framework that integrates DeepInsight-transformed gene expression, longitudinal laboratory trajectories, and treatment history to significantly outperform existing baseline methods in predicting residual overall survival for multiple myeloma patients across various post-diagnosis timepoints.

The Gist

Imagine you are trying to predict how long a car will last.

The Old Way (Current Medical Standard):
Right now, doctors look at a car the moment you buy it. They check the engine type, the mileage, and the color. Based on that single snapshot, they put the car into a "High Risk" or "Low Risk" bucket. Once that bucket is chosen, it stays the same forever. Even if the car starts making a weird noise three years later, or if you change the oil regularly, the original prediction doesn't update. It's static.

The New Way (This Paper's Solution):
The researchers in this paper built a "Smart GPS" for Multiple Myeloma (a type of blood cancer). Instead of just looking at the car when you buy it, this GPS watches the car every single month for the first 18 months.

Here is how it works, broken down into simple parts:

1. The Three "Sensors" (Data Sources)

The model doesn't just look at one thing; it combines three different streams of information, like a detective gathering clues from three different witnesses:

• The DNA Blueprint (Gene Expression): Think of this as a high-resolution photo of the car's engine. The researchers turned thousands of genetic data points into a single image (using a trick called DeepInsight). This helps the computer "see" patterns in the cancer's biology that a simple list of numbers would miss.
• The Dashboard Gauges (Longitudinal Labs): This is the most important part. Just like a car's dashboard shows speed, fuel, and temperature changing over time, this model tracks 10 different blood tests (like hemoglobin and kidney function) month after month. It doesn't just see the number; it sees the trend. Is the fuel tank draining faster? Is the engine getting hotter?
• The Repair Log (Treatment History): This tracks what medicines the patient has taken. Did they try a new oil change? Did they switch to a different fuel type? The model learns how the car reacts to these repairs.

2. The "Smart Fusion" (How it Thinks)

The real magic is in how the computer combines these clues.

• The Problem: Sometimes a patient misses a blood test, or the doctor forgets to record a pill. In the old days, the computer would get confused or just guess.
• The Solution: This model has a "Gatekeeper." It knows when data is missing. If a blood test is missing, it doesn't panic; it just relies more heavily on the DNA photo and the treatment log. It's like a detective who knows, "Okay, we don't have the witness's statement today, but the fingerprint and the security camera footage are still strong."

3. The "Dynamic" Update

This is the biggest game-changer.

• Month 1: The model gives a prediction based on the initial diagnosis.
• Month 6: The model looks at the last 6 months of blood tests and treatments. It updates the prediction. Maybe the patient is doing better than expected! The risk score drops.
• Month 12: It updates again.
• Why it matters: In the old system, if a patient's condition worsened, the doctor had to wait for a new "staging" system to re-evaluate them. This model updates the risk score continuously, giving a fresh, accurate forecast at any moment during the first 18 months.

4. The "Teacher and Student" Trick

The researchers built a super-smart "Teacher" model that uses all three data sources (DNA, Labs, and Meds). But they knew that in the real world, some hospitals might not have all that data (maybe they only have the DNA photo and basic blood work).

So, they created a "Student" model. They taught the Student everything the Teacher knew, but forced the Student to learn using only the DNA photo and basic blood work.

• The Result: Even without the full treatment history or complex lab trends, the "Student" model was still very good at predicting survival. This means the technology can be used in smaller hospitals that don't have massive databases, making it much more useful for regular people.

5. Did it Work?

• The Score: In tests, the new model scored 0.77 (where 1.0 is perfect and 0.5 is a coin flip). The old standard methods only scored around 0.63.
• The Proof: When they split patients into "High Risk" and "Low Risk" groups, the model was incredibly accurate at separating them. The "High Risk" group had a much higher chance of survival issues, and the "Low Risk" group did much better.
• The "Why": The researchers even asked the model why it made its decisions. It pointed to known biological reasons (like specific genes related to how cells handle stress) and known medical facts (like how albumin levels drop when the disease gets worse). This proves the model isn't just guessing; it's actually "understanding" the biology.

The Bottom Line

This paper introduces a living, breathing prediction tool. Instead of freezing a patient's fate at the moment of diagnosis, it watches their journey, learns from their changing blood work and treatments, and constantly updates the forecast. It's like switching from a static paper map to a real-time GPS that reroutes you based on traffic, accidents, and road conditions as they happen.

Technical Summary

1. Problem Statement

Multiple Myeloma (MM) prognosis is currently reliant on static staging systems (e.g., ISS, R-ISS) that assign patients to fixed risk categories at diagnosis. These systems fail to incorporate:

• Longitudinal Dynamics: The evolving disease burden and treatment response captured by serial laboratory measurements over time.
• Treatment Context: The impact of specific drug regimens (e.g., proteasome inhibitors, immunomodulatory drugs) administered during the multi-phase treatment course.
• Temporal Flexibility: Existing computational models often rely on single-time-point snapshots, making them unable to update risk estimates as new clinical data becomes available during therapy.

The authors aim to develop a dynamic, multimodal deep learning framework that predicts residual overall survival at any point within the first 18 months post-diagnosis by integrating gene expression, longitudinal laboratory trajectories, and treatment history.

2. Methodology

A. Data Sources and Cohorts

• Primary Cohort (Development): MMRF CoMMpass study ($n=752$). Includes baseline bulk RNA-seq, serial laboratory measurements (10 analytes over 18 months), and treatment records.
• External Cohort (Validation): GSE24080 ($n=507$). A microarray-based cohort with only baseline gene expression and 5 summary clinical variables (no longitudinal or treatment data).
• Prediction Paradigm: A dynamic landmark framework. For any prediction time $t \in [1, 18]$ months, the model uses all data observed up to $t$ to predict survival risk for the subsequent horizon.

B. Data Modalities & Preprocessing

1. Gene Expression (DeepInsight):
   • 5,000 high-variance genes were selected.
   • Transformed into $96 \times 96$ single-channel images using DeepInsight. This technique uses t-SNE to map co-expressed genes to spatially proximate pixels, preserving local feature relationships and enabling Convolutional Neural Networks (CNNs) to exploit spatial co-expression structures.
2. Longitudinal Laboratories:
   • 10 analytes (e.g., $\beta2M$, LDH, FLCs, Albumin) binned into monthly intervals.
   • Missingness Modeling: Instead of imputation, the model explicitly handles irregular sampling using binary observation masks, time-since-last-observation (TSLO) features, and "any-observation" flags.
3. Treatment History:
   • Encoded as binary indicators for three drug classes (Bortezomib, Carfilzomib, IMiDs) on the same monthly grid.

C. Model Architecture

The model employs a late-fusion multimodal architecture with a Cox Proportional Hazards (PH) head:

• Encoders:
  • Gene Encoder: A lightweight CNN (5 blocks) processes the DeepInsight image.
  • Lab Encoder: A dual-stream Transformer. One stream processes values/masks; a parallel stream encodes missingness patterns.
  • Drug Encoder: A Transformer processes drug usage grids.
• Gated Fusion Mechanism:
  • Modality embeddings are projected to a shared dimension.
  • A gate network (MLP) conditioned on the time embedding and missingness summaries generates softmax weights to dynamically fuse the modalities. This allows the model to down-weight sparse or outdated modalities.
  • Regularization: Includes auxiliary per-modality Cox heads (to prevent modality collapse) and modality dropout (simulating external deployment by zeroing clinical data during training).
• Loss Function: Combines the main fusion-level Cox partial log-likelihood with auxiliary losses from individual modality heads.

D. Training Strategy

• Sampling Contract: To address small sample size ($n=624$) and non-uniform event density:
  • Each patient contributes exactly $K=3$ landmark samples per epoch.
  • Landmark time $t$ is sampled from a progressively broadened distribution (focusing on early, event-rich time points initially).
• Knowledge Distillation: A "Student" model was distilled from the full "Teacher" model. The student retains only the DeepInsight image encoder and baseline clinical features, trained to mimic the teacher's outputs. This enables deployment on external cohorts lacking longitudinal data.

3. Key Contributions

1. Dynamic Multimodal Framework: First application of a gated fusion architecture that integrates high-dimensional gene expression (via DeepInsight), irregular longitudinal labs, and treatment history for MM survival prediction.
2. DeepInsight for Survival: Demonstrates that transforming tabular gene expression into spatial images improves prognostic discrimination over standard MLPs by capturing co-expression modules.
3. Missingness-Aware Design: Explicitly models clinical data irregularity (missingness) rather than treating it as noise, allowing the model to distinguish between "true absence of signal" and "missing observation."
4. Distillation for Deployment: Successfully compressed the complex multimodal teacher into a reduced-input student model that maintains predictive power on external cohorts with limited data modalities.

4. Results

Performance Metrics (CoMMpass Cohort)

• Cross-Validation (5-fold):
  • C-index: $0.773 \pm 0.024$
  • 1-year Time-dependent AUC (tdAUC1yr): $0.789 \pm 0.021$
• Benchmarking: Outperformed all baselines, including DeepSurv ($0.633$), Random Survival Forests ($0.636$), and Elastic Net.
• Landmark Analysis: Performance improved as more longitudinal data accumulated (C-index ranged $0.73–0.90$ across months 1–18).
• Risk Stratification: Significant separation between high/low-risk groups at 6 and 12 months (Log-rank $p < 0.001$; Hazard Ratios $3.46–3.93$).

Ablation Studies

• Longitudinal Labs: The strongest individual contributor (C-index $0.693$).
• DeepInsight vs. MLP: The spatial encoding (DeepInsight) outperformed a standard MLP on the same genes ($0.624$ vs. $0.596$).
• Full Model: The multimodal fusion provided a significant gain ($\Delta C \approx +0.08$) over the best single modality.

External Validation (GSE24080)

• The distilled student model (using only DeepInsight + 5 clinical features) achieved:
  • C-index: $0.672$
  • tdAUC1yr: $0.740$
• This outperformed all transferred baseline models, demonstrating the robustness of the learned gene representation across platforms.

Interpretability

• Genes: Identified high-risk associations with Ubiquitin-Proteasome pathway genes (e.g., UBE2Q1), ER stress markers (P4HB), and Interferon Alpha Response pathways, consistent with MM biology.
• Temporal: Heatmaps confirmed that higher levels of LDH, $\beta2M$, and FLCs increase risk, while Albumin and Hemoglobin are protective, aligning with established staging systems.

5. Significance

This study presents a paradigm shift in MM prognostication from static, diagnosis-only models to dynamic, time-adaptive frameworks. By effectively fusing heterogeneous data sources and handling real-world clinical data irregularities (missingness), the model provides more accurate and updatable risk estimates. The successful distillation of the model for external deployment suggests a viable path for integrating advanced AI prognostics into clinical settings where only baseline data may be available, bridging the gap between high-resource research cohorts and routine clinical practice.