🤖 machine learning

SurvBench: A Standardised Preprocessing Pipeline for Multi-Modal Electronic Health Record Survival Analysis

SurvBench is an open-source, standardized preprocessing pipeline that converts raw PhysioNet electronic health record data into model-ready tensors across four critical-care databases and four modalities, addressing the inconsistency in upstream data processing that currently hinders fair comparison of deep-learning survival models.

Original authors: Munib Mesinovic, Tingting Zhu

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Munib Mesinovic, Tingting Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build the ultimate recipe for predicting how long a patient will stay in the hospital or if they might pass away. You have four different kitchens (four massive databases of patient records: MIMIC-IV, eICU, HiRID, and MC-MED). Each kitchen has its own way of storing ingredients, its own measuring cups, and its own messy counters.

In the past, every chef (researcher) who wanted to test their new recipe (AI model) had to clean their own kitchen first. One chef might chop the vegetables (data) into tiny squares, while another leaves them whole. One might wash the meat (impute missing data) with water, another with soap. Because everyone cleaned their kitchen differently, it was impossible to tell if a new recipe was actually better, or if it just worked better because that chef had a cleaner kitchen to start with.

Enter SurvBench.

Think of SurvBench as a universal, automated kitchen prep station. It's a free, open-source tool that takes the raw, messy ingredients from those four different hospitals and turns them into perfectly standardized, "model-ready" packages.

Here is how it works, broken down into simple parts:

1. The Universal Translator (Preprocessing)

The paper argues that the biggest problem in this field isn't the AI models themselves, but the "preprocessing gap." SurvBench fixes this by acting as a strict translator.

Standardizing Units: If one hospital records temperature in Fahrenheit and another in Celsius, SurvBench converts them all to the same scale before anyone touches them.
Handling Missing Data: Sometimes a patient's heart rate wasn't recorded for an hour. SurvBench doesn't just guess; it fills in the gap using a specific rule (looking at the last known value) and, crucially, it attaches a "missingness mask." This is like a sticky note on the ingredient that says, "This value was filled in by the machine, not measured." This ensures the AI knows the difference between a real zero and a guessed zero.
Time Windows: Instead of looking at a chaotic stream of data, SurvBench chops the patient's stay into neat, equal-sized time blocks (like slicing a loaf of bread), so every patient's data looks the same shape.

2. The Four Kitchens (The Databases)

SurvBench connects to four major public databases, which are like four different types of hospitals:

MIMIC-IV & eICU: These are huge collections of Intensive Care Unit (ICU) records. They track patients from the moment they enter the ICU until they leave or pass away.
HiRID: Another ICU database, but from Switzerland, with very high-resolution data (measurements taken every few minutes).
MC-MED: This is an Emergency Department (ED) database. Instead of just tracking death, it tracks where a patient goes next: do they go home, go to a regular hospital bed, or get sent to the ICU? This is a "competing risks" scenario, like a race where there are multiple possible finish lines.

3. The Ingredients (Modalities)

SurvBench doesn't just look at one thing; it combines four different types of "ingredients" into a single tensor (a multi-dimensional data block):

Static: Things that don't change, like age, sex, and weight.
Time-Series: The heartbeat, blood pressure, and lab results that change over time.
ICD Codes: The medical diagnosis codes (like a patient's history of diseases).
Radiology Reports: The text from X-ray and CT scan reports, which the tool turns into mathematical numbers using advanced language models.

4. The Taste Test (The Results)

To prove their kitchen prep station works, the authors didn't just build the station; they cooked five different "recipes" (AI models) using the standardized ingredients to see which one tasted best.

They tested classic statistical methods and modern deep learning models (like Transformers, which are usually used for language but are now being used for time-series data).
The Winner: On the ICU data, models that could look at the history of the patient's vitals (time-series) performed much better than those that only looked at static facts like age.
The Multi-Modal Star: A new model they built called TransformerSurv (which uses all four ingredient types) performed very well, especially on the Emergency Department data where it could combine lab results with the text from radiology reports.

The Big Takeaway

The paper claims that SurvBench is the first tool to standardize the "prep work" for survival analysis across these major databases.

Before this, comparing two AI models was like comparing two runners who started at different starting lines. Now, with SurvBench, everyone starts at the exact same line, with the same shoes, and the same track. This allows researchers to finally say, "Model A is actually better than Model B," rather than, "Model A just had a better data cleaning process."

The authors emphasize that this tool is not a magic cure-all or a finished product ready for hospitals to use tomorrow. It is a foundation. It provides a fair playing field so that future scientists can build better, safer, and more reliable AI models without getting bogged down in the messy details of cleaning the data.

In short: SurvBench is the great equalizer. It takes the messy, inconsistent data from different hospitals, cleans it up with the exact same rules, and hands it to AI researchers so they can finally compare their ideas fairly.

Technical Summary: SurvBench

Problem Statement

Deep learning survival models for Electronic Health Record (EHR) data face a significant barrier to reproducibility and fair comparison: the "preprocessing gap." While downstream modeling architectures (e.g., DeepSurv, Dynamic-DeepHit) are increasingly sophisticated, the upstream steps required to convert raw EHR exports into model-ready inputs are rarely shared, documented, or standardized. Critical decisions regarding cohort definition, time discretization, missingness handling, feature filtering, and censoring rules vary between studies. Consequently, reported performance differences (e.g., in concordance indices) often reflect these preprocessing choices rather than genuine modeling contributions. Furthermore, existing benchmarks like MIMIC-III are superseded by MIMIC-IV and lack support for modern survival analysis tasks, including competing risks and multi-modal inputs.

Methodology

The authors present SurvBench, an open-source, configurable preprocessing pipeline designed to standardize the creation of survival analysis cohorts from raw PhysioNet exports.

Core Pipeline Architecture

SurvBench processes data from four critical-care databases: MIMIC-IV, eICU, MC-MED, and HiRID. It supports four input modalities:

Static features: Demographics (age, sex, etc.).
Time-series: Vital signs and laboratory values.
ICD codes: Diagnosis histories (multi-hot encoded).
Radiology: Free-text report embeddings (via HuggingFace models like Clinical-Longformer).

The pipeline operates through the following stages:

Cohort Definition: Configurable via YAML. It enforces patient-level splitting to prevent data leakage (ensuring all stays for a single patient remain in one fold). It applies specific filters per dataset (e.g., 24-hour minimum length of stay for ICU datasets, age filters).
Survival Label Processing: Supports both single-risk (e.g., in-ICU mortality) and competing-risks (e.g., ED disposition: admission, observation, ICU admission) endpoints. Censoring is handled via administrative truncation at a configurable horizon (e.g., 240 hours for ICU, 24 hours for ED).
Time-Series Processing: Irregularly sampled measurements are aggregated into fixed-width temporal windows (e.g., 4-hour bins) using mean aggregation. Features are filtered based on a missingness threshold (default 1% of training windows).
Missingness Handling: A leakage-safe, two-stage strategy is employed:
1. Forward-fill: Within-patient forward filling at hourly resolution.
2. Mean Imputation: Remaining missing values are filled with the per-feature mean calculated only on the training fold.
3. Masking: A binary mask is generated alongside every tensor to distinguish observed values from imputed ones.
Scaling: Z-score normalization is applied using StandardScaler fitted exclusively on the training fold. Dynamic and static feature blocks are scaled independently to prevent static features from dominating the scaling statistics of dynamic blocks.
Multi-Modal Integration: Static features are broadcast across all time windows. ICD codes and radiology embeddings are concatenated with the time-series tensor.

Reference Baselines

To validate the pipeline and provide a benchmark, the authors trained five models on the generated cohorts:

Cox Proportional Hazards: Fitted on mean-pooled static features.
DeepHit: A discrete-time competing-risks network using static features.
Dynamic-DeepHit: Extends DeepHit with an LSTM and temporal attention over time-series data.
DySurv: An LSTM-variational autoencoder with cause-specific logistic-hazard heads.
TransformerSurv: A new architecture utilizing a pre-LN Transformer encoder. It was tested in two configurations: (a) time-series + static, and (b) full multi-modal (including ICD and radiology).

Key Results

The pipeline generated cohorts ranging from ~15,000 (HiRID) to ~128,000 (eICU) ICU stays. Key findings from the evaluation of reference baselines include:

Importance of Temporal Data: Static-only models (Cox, DeepHit) significantly underperformed compared to models utilizing time-series data. On single-risk ICU tasks, temporal models achieved concordance ( $C_{td}$ ) scores of 0.82–0.86, whereas static models scored 0.61–0.64.
Multi-Modal Benefits: On the MC-MED dataset (which includes ICD and radiology), the multi-modal TransformerSurv achieved the highest concordance ( $C_{td} = 0.774$ ) and improved calibration metrics (Integrated Brier Score and Negative Binomial Log-Likelihood) compared to the time-series-only variant.
Cross-Dataset Transfer: In an external validation task training on eICU and testing on MIMIC-IV, the Transformer model achieved the highest out-of-distribution discrimination ( $C_{td} = 0.812 \pm 0.007$ ), outperforming Dynamic-DeepHit and DySurv.
Calibration vs. Discrimination: The study highlighted a trade-off where models could rank patients well (high concordance) while producing miscalibrated probabilities. For instance, on MC-MED, Cox proportional hazards had the best calibration (lowest IBS) but the worst concordance.

Significance and Claims

The authors position SurvBench not as a provider of a "state-of-the-art" model, but as a standardized infrastructure to enable fair comparison.

Reproducibility: By providing a single, end-to-end YAML-controlled pipeline, SurvBench eliminates the "preprocessing gap" where performance variations are often driven by undocumented data handling choices rather than model architecture.
Multi-Modal Support: It is the first public pipeline to support the integration of time-series, static, ICD, and radiology modalities for survival analysis across multiple major EHR databases (MIMIC-IV, eICU, MC-MED, HiRID).
Leakage Safety: The pipeline strictly enforces that imputation, scaling, and feature filtering are fit only on the training fold, addressing a common source of data leakage in survival analysis.
Foundation for Future Work: The authors state that the provided reference baselines and processed cohorts serve as a "substrate" against which future deep learning methods can be measured under matched preprocessing conditions.

The paper concludes that while the benchmark inherits demographic limitations from its source data (primarily US and Swiss populations), it provides a rigorous technical foundation for model comparison, allowing researchers to focus on algorithmic innovation rather than data engineering.