SurvHTE-Bench: A Benchmark for Heterogeneous Treatment Effect Estimation in Survival Analysis

This paper introduces SurvHTE-Bench, the first comprehensive benchmark for evaluating heterogeneous treatment effect estimation in survival analysis, featuring a diverse suite of synthetic, semi-synthetic, and real-world datasets to enable rigorous, fair, and reproducible comparisons of causal survival methods.

The Gist

Imagine you are a doctor trying to decide which of two new medicines works best for a patient. In a perfect world, you could give Medicine A to one patient, wait to see what happens, then give Medicine B to the same patient and compare the results. But life isn't like that. You can't go back in time, and in real-world medical data, patients often drop out of studies, move away, or stop taking their meds before the study ends. This is called censoring.

Furthermore, Medicine A might work miracles for a young athlete but do nothing for an elderly person with other health issues. This difference in how treatments work for different people is called Heterogeneous Treatment Effects (HTE).

The problem is: How do you figure out exactly who benefits from which medicine when your data is messy, incomplete, and full of "dropouts"?

This is the challenge tackled by the paper "SURVHTE-BENCH." Think of this paper as the creation of the ultimate "Flight Simulator for Medical Decision Makers."

Here is a simple breakdown of what they did:

1. The Problem: The "Missing Puzzle Pieces"

In survival analysis (studying how long people live or stay healthy), researchers often have to guess what would have happened to a patient who dropped out of a study.

• The Analogy: Imagine trying to judge a race, but half the runners quit halfway through. You don't know if they would have won, lost, or finished in the middle. If you try to guess the winner based only on the runners who finished, you might get it wrong.
• The Issue: There are many different computer algorithms (methods) trying to solve this guessing game, but they all test themselves on different, tiny, made-up datasets. It's like every driver testing their car on a different, private track. No one knows which car is actually the best for the real world.

2. The Solution: SURVHTE-BENCH (The Ultimate Test Track)

The authors built a massive, standardized testing ground called SURVHTE-BENCH. It's the first time anyone has created a "big league" stadium to test all these different algorithms fairly.

They built three types of test tracks:

• The "Video Game" Tracks (Synthetic Data):
  They created 40 different computer-generated worlds. In these worlds, they know the exact truth (like the answer key to a test). They can set the rules to be easy (everyone finishes the race) or incredibly hard (runners quit because they are sick, or the race is rigged). This lets them see which algorithms break when the rules get tough.

  • Analogy: This is like a flight simulator where the pilot can turn on "turbulence," "engine failure," or "fog" to see if the autopilot can handle it.

• The "Real-World Dress Rehearsal" (Semi-Synthetic Data):
  They took real patient data (like medical records from an ICU) but swapped out the treatment and outcome with simulated ones. This keeps the messy, real-life complexity of human bodies but gives them a known answer key.

  • Analogy: Taking a real car and driving it on a closed course where you know exactly how the brakes should work, so you can test if the driver's instincts are right.

• The "Real Race" (Real Data):
  They tested the algorithms on two actual famous datasets: one about twins (where we actually do know the answer because we have two people to compare) and one about HIV treatments.

3. The Contenders: The "Race Cars"

They didn't just test one method; they lined up 53 different algorithms (the race cars). They grouped them into three main families:

1. The "Imputers": These try to fill in the missing puzzle pieces first (guessing when the dropouts would have finished) and then run a standard analysis.
2. The "Direct Survivors": These are built specifically to handle the "dropouts" without guessing first. They understand the rules of the survival game natively.
3. The "Meta-Learners": These are smart coaches that combine different strategies to find the best approach for the specific situation.

4. The Results: Who Won?

The big takeaway is that there is no single "best" car for every track.

• On easy tracks (where few people drop out and the study is perfectly randomized), the "Imputers" (like Double-ML) were very fast and accurate.
• On hard tracks (where many people drop out, or the study is messy and biased), the "Direct Survivors" and "Meta-Learners" (specifically things like Causal Survival Forests and S-Learner-Survival) were the champions. They were more robust and didn't crash as easily.

The "Aha!" Moment:
The paper found that if you are dealing with messy, real-world data where people drop out frequently (high censoring), you shouldn't use the old-school methods that try to "fix" the data first. You need methods that are designed to handle the messiness from the start.

Why Does This Matter?

Before this paper, a doctor or policy-maker might pick a method because it looked good in one small study, only to find out it fails miserably when applied to a real hospital with thousands of patients.

SURVHTE-BENCH is like a standardized "Consumer Reports" for medical AI. It tells us:

• "If your data is clean, use Method A."
• "If your data is messy and people drop out, use Method B."
• "If you ignore the dropouts, you might make dangerous mistakes."

By providing this open-source "test track," the authors hope to stop researchers from reinventing the wheel and help the medical community build tools that actually save lives, rather than just looking good on paper.

Technical Summary

Here is a detailed technical summary of the paper "SURVHTE-BENCH: A Benchmark for Heterogeneous Treatment Effect Estimation in Survival Analysis."

1. Problem Statement

The paper addresses the critical challenge of estimating Heterogeneous Treatment Effects (HTEs) in right-censored survival data. While estimating population-level Average Treatment Effects (ATE) is common, precision medicine and individualized policy-making require understanding how treatment effects vary across individuals (Conditional Average Treatment Effects, or CATE).

Estimating CATE in survival settings is uniquely difficult due to:

• Right-censoring: The event of interest (e.g., death, relapse) is not observed for all subjects within the study period.
• Unobserved Counterfactuals: Only one potential outcome (treated or control) is observed per unit.
• Complex Identification Assumptions: Standard causal assumptions (Ignorability, Positivity, Consistency) must hold, and an additional assumption of Ignorable Censoring is required. In practice, these assumptions are often violated (e.g., informative censoring due to dropout, unmeasured confounding, or lack of overlap).

Despite recent methodological advances (e.g., Causal Survival Forests, Survival Meta-Learners), there is no standardized benchmark to evaluate these methods. Existing evaluations rely on bespoke simulations with inconsistent assumptions, making it impossible to compare robustness or determine which methods perform best under specific real-world conditions.

2. Methodology: SURVHTE-BENCH

The authors introduce SURVHTE-BENCH, the first comprehensive benchmark designed to systematically evaluate survival HTE estimators. The benchmark consists of three main components:

A. Synthetic Datasets (40 Datasets)

The core of the benchmark is a modular suite of 40 synthetic datasets created by crossing 8 Causal Configurations with 5 Survival Scenarios.

• Causal Configurations: These systematically vary the validity of identification assumptions:
  • RCT: Randomized Controlled Trials (50% and 5% treatment rates).
  • OBS-CPS: Observational studies with correctly specified propensity scores (Ignorability holds).
  • OBS-UConf: Unobserved confounding (Ignorability violated).
  • OBS-NoPos: Lack of positivity (treatment assignment is deterministic for some covariates).
  • InfC: Informative censoring (censoring depends on the event time, violating ignorable censoring).
  • Combinations of the above (e.g., Unobserved Confounding + Informative Censoring).
• Survival Scenarios: These vary the underlying data-generating process:
  • Distributions: Cox Proportional Hazards, Accelerated Failure Time (AFT), and Poisson hazards.
  • Censoring Rates: Low (<30%), Medium (30-70%), and High (>70%).
• Ground Truth: Since the data is synthetic, the true CATE (Restricted Mean Survival Time difference) is known for every unit, allowing for precise error calculation.

B. Semi-Synthetic Datasets (10 Datasets)

To bridge the gap between synthetic control and real-world complexity, the authors pair real-world covariates with simulated treatments and outcomes:

• ACTG: Based on the HIV clinical trial (23 covariates), simulating moderate censoring (~51%) and realistic treatment imbalance.
• MIMIC-IV: Derived from ICU records (36 covariates), creating 9 variants with varying censoring rates (53% to 88%) and mechanisms (linear vs. non-linear, independent vs. confounded treatment assignment).

C. Real-World Datasets (2 Datasets)

• Twins Dataset: A unique dataset where twin pairs allow for "ground truth" CATE estimation (one twin is treated, the other is the control, observing both outcomes).
• ACTG 175: A real HIV clinical trial without ground truth, used to test robustness by artificially injecting high levels of censoring (>90%) and observing how estimates shift.

D. Evaluated Methods (53 Variants)

The benchmark unifies and implements 53 estimators across three families:

1. Outcome Imputation Methods: Impute censored times (using Pseudo-observations, Margin, or IPCW) and apply standard CATE estimators (S-, T-, X-, DR-Learners, Double-ML, Causal Forest) with various base learners (Lasso, RF, XGBoost).
2. Direct-Survival CATE Methods: Methods designed natively for censored data (Causal Survival Forests, SurvITE).
3. Survival Meta-Learners: Adaptations of S-, T-, and Matching-Learners using survival-specific base models (Random Survival Forests, DeepSurv, DeepHit).

3. Key Contributions

1. Method Unification: Provides the first modular framework categorizing and implementing 53 survival HTE methods, facilitating fair comparison.
2. Systematic Stress-Testing: Moves beyond simple simulations to test methods under simultaneous assumption violations (e.g., unobserved confounding + informative censoring) and diverse survival dynamics.
3. Comprehensive Evaluation Protocol: Establishes metrics for CATE RMSE, ATE bias, and auxiliary tasks (imputation accuracy, survival fit) across synthetic, semi-synthetic, and real data.
4. Open Resources: Releases code and datasets (including the synthetic suite and semi-synthetic generators) to the community to ensure reproducibility.

4. Key Results and Findings

The benchmark reveals that no single method dominates across all settings; performance is highly context-dependent.

• Impact of Censoring:
  • In low-censoring randomized settings, outcome imputation methods (e.g., Double-ML, X-Learner) often excel.
  • As censoring rates increase (especially >70%), Survival Meta-Learners (specifically S-Learner-Survival and Matching-Survival) and Causal Survival Forests consistently outperform imputation-based methods. Direct survival modeling handles the uncertainty of heavy censoring better than imputing missing times.
• Robustness to Assumption Violations:
  • Unobserved Confounding: Survival meta-learners and Causal Survival Forests maintain relatively stable performance and lower bias compared to standard meta-learners.
  • Informative Censoring: This is the most challenging scenario. All methods degrade, but survival-aware approaches (Causal Survival Forests, Matching-Survival) show superior robustness compared to outcome imputation methods.
  • Positivity Violations: Methods relying on overlap (like X-Learner) struggle, while Matching-based approaches show resilience.
• Component Analysis:
  • Imputation: The "Margin" imputation strategy generally provided the best balance of accuracy and robustness under heavy censoring.
  • Base Learners: For survival meta-learners, DeepSurv (a deep learning survival model) consistently yielded the best results, outperforming Random Survival Forests in many high-dimensional scenarios.
• Real-World Insights:
  • On the Twins dataset, S-Learner-Survival and S-Learner (with imputation) performed best.
  • On the ACTG HIV trial, methods showed distinct behaviors under high censoring: Causal Survival Forests produced stable estimates, while survival meta-learners (especially T-Learner and Matching) exhibited high variance, suggesting sensitivity to censoring conditions in real data.

5. Significance

• Standardization: SURVHTE-BENCH fills a critical gap in causal survival analysis by providing a standardized, reproducible benchmark, replacing fragmented, author-specific evaluations.
• Practical Guidance: It offers actionable guidance for practitioners:
  • Use Double-ML or X-Learner for low-censoring, well-overlapped data.
  • Switch to Survival Meta-Learners (S-Learner/Matching with DeepSurv) or Causal Survival Forests when dealing with high censoring, informative censoring, or unobserved confounding.
• Future Research: The modular design allows for future extensions to time-varying treatments, instrumental variables, and different estimands (e.g., median survival time), fostering continued progress in the field of causal survival inference.

In summary, the paper demonstrates that while standard causal inference methods can be adapted for survival data, explicitly modeling the survival structure (rather than just imputing outcomes) is crucial for robustness in realistic, high-stakes scenarios characterized by heavy censoring and assumption violations.