A Domain Incremental Continual Learning Benchmark for ICU Time Series Model Transportability

Technical Summary: A Domain Incremental Continual Learning Benchmark for ICU Time Series Model Transportability

Problem Statement
Machine learning models for clinical outcome prediction have demonstrated increasing accuracy but face significant barriers to deployment in smaller hospitals. These institutions often lack the resources for data collection, labeling, and computational power required to train models from scratch. While transferring models trained on large hospital datasets (e.g., MIMIC-III) to smaller institutions offers a solution, these models are typically trained and validated on single-hospital data, raising concerns about generalizability. The authors identify substantial differences in measurement distributions and frequencies across different regions of the United States. When models are transferred without adjustment, these distribution shifts can degrade performance. Furthermore, simply retraining on new data sources leads to "catastrophic forgetting," where the model loses knowledge of the original domain. The paper frames this challenge as a Domain Incremental Learning (DIL) problem: the task (patient outcome prediction) remains constant, but the input data distribution varies across regions, requiring a model to adapt to new domains while retaining features from previous ones.

Methodology
The authors propose a DIL benchmark and evaluate specific methods to mitigate catastrophic forgetting in a clinical setting.

• Benchmark Framework: The study initializes a model on the MIMIC-III dataset (Beth Israel Deaconess Medical Center) and transfers it to four distinct US regions using the eICU dataset: South, Midwest, West, and Northeast. The evaluation covers four clinical tasks: In-Hospital Mortality (IHM), Decompensation, Length-of-Stay (LOS), and Phenotyping.
• Data Preprocessing: The authors utilize 17 clinical measurements (e.g., vitals, lab results, Glasgow Coma Score). Data is processed to ensure consistent hourly intervals, with missing values imputed via forward filling or normal values. An imputation indicator is added for each feature. Patient stays are filtered to include only adults (≥18) with single ICU admissions and specific minimum stay durations depending on the task.
• Model Architecture: Standard BiLSTM models are used for IHM and Phenotyping, while LSTM models are used for LOS and Decompensation.
• Proposed Methods: The paper evaluates and compares three approaches against a baseline (no DIL):
  1. Data Replay: Stores a memory buffer of samples from previous sources to be replayed during training on new sources. The authors note that traditional replay can lead to overfitting if the buffer is small or unrepresentative.
  2. Elastic Weight Consolidation (EWC): A regularization method that penalizes changes to parameters deemed important for previous tasks, calculated using the Fisher Information Matrix.
  3. Proposed Combined Method: A novel approach that integrates EWC with a modified data replay strategy. This modification adjusts the frequency of updates using data from previous sources. Instead of progressive weighting that may overfit to small buffers, the authors propose a mechanism where replay loss is applied at even intervals, ensuring each sample in the buffer is seen at most once per training cycle. This is combined with EWC regularization to preserve critical weights.

Key Contributions

1. DIL Benchmark for Clinical Time Series: The authors introduce a benchmark that tests model transferability from a large hospital system (MIMIC-III) to various US regions (eICU). They provide a detailed analysis of the distribution shifts in features and measurement frequencies across these regions.
2. Novel DIL Method: They propose a combined method that merges EWC with an adjusted data replay technique. This method introduces the ability to control the frequency of updates using historical data to minimize overfitting risks associated with limited memory buffers in clinical settings.
3. Comprehensive Evaluation: The study evaluates four methods (Baseline, EWC, Traditional Replay, Adjusted Replay, and the Combined Method) across four distinct clinical tasks, including binary classification, multi-label classification, and sequence-to-sequence tasks.

Results
Experimental results indicate that the proposed combined method generally outperforms EWC and data replay used in isolation, particularly for non-sequence-to-sequence tasks (IHM and Phenotyping).

• Performance: For IHM and Phenotyping, the combined method demonstrated equal or superior performance across all eICU regions compared to other methods.
• Task Variability: While the combined method showed significant improvements over the baseline for Phenotyping, the gains were less pronounced for IHM, which the authors attribute to the task's simplicity and strong correlation between MIMIC-III and eICU regions.
• Sequence-to-Sequence Challenges: For LOS and Decompensation, the authors observed a significant drop in performance when transferring from MIMIC-III to eICU regions, despite improvements over the baseline. This highlights the inherent difficulty of sequence-to-sequence tasks under domain shift.
• Regional Differences: The results underscore that the optimal method can vary by region and task. For instance, in the Northeast region for Decompensation, the baseline sometimes outperformed DIL methods due to high correlation with the source data.

Significance and Limitations
The paper claims significance in addressing the feasibility of deploying robust machine learning models in smaller hospitals by providing a framework to assess and improve model transportability. It highlights that models must adapt to regional variations in data collection practices and patient demographics to maintain performance.

The authors modestly note specific limitations and future directions:

• Privacy Constraints: The proposed DIL methods (replay and EWC) still require some patient data to be passed between clinics, which may be restricted by privacy regulations. They suggest future work could focus on "memory-less" transfer methods.
• Clinical Shortcomings: The study underscores specific shortcomings in model transferability within clinical settings, indicating that while the proposed method improves performance, further research is needed to fully solve the generalizability problem.
• Scope: The evaluation is limited to two sequential sources (MIMIC-III and one eICU region) in the primary experiments, though the authors expect the performance gap between their combined method and traditional DIL methods to grow with an increasing number of sources.

Ultimately, the work aims to facilitate the adoption of predictive tools in resource-constrained environments by demonstrating that models can be adapted to new domains while retaining prior knowledge, provided the right incremental learning strategies are employed.