Can Physician Expertise Improve Machine Learning Identification of Delirium?

Technical Summary: Can Physician Expertise Improve Machine Learning Identification of Delirium?

Problem Statement
Delirium, an acute neuropsychiatric condition affecting up to 50% of elderly hospitalized patients, is frequently missed in routine care. Untreated delirium leads to severe outcomes, including doubled healthcare costs, increased mortality, and long-term cognitive decline. While preventable in up to 40% of cases through multi-component programs, these interventions are underutilized due to a lack of standardized protocols and scalable early detection tools. Existing machine learning (ML) prediction models often suffer from limitations such as reliance on non-routinely collected data (e.g., cognitive assessments), low accuracy, neglect of unstructured text data, and susceptibility to model drift. Furthermore, standard automated ML (aML) approaches often lack transparency ("black-box" nature), hindering clinical trust and adoption.

Methodology: User-Centered Interactive Machine Learning (UC-iML)
The authors propose a User-Centered Interactive Machine Learning (UC-iML) framework designed to integrate physician expertise directly into the ML development lifecycle. Unlike traditional workflows where clinicians only review final models, this framework employs a "human-in-the-loop" approach across three iterative stages:

1. Requirements Acquisition (RA): Physicians (acting as "Machine Teachers") collaborate with developers to define clinical priorities. They prioritize features spanning administrative data (e.g., length of stay, discharge disposition), clinical signals (e.g., lab abnormalities, ICU admission), and medication patterns. The goal is to prioritize high sensitivity for screening rather than relying solely on accuracy, which can be misleading in imbalanced datasets.
2. Design and Prototyping (DP): This stage involves multi-source feature engineering and physician-guided refinement. The system integrates:
   • Administrative & Clinical Data: ICD-10 codes, vital signs, and lab results (e.g., sodium, creatinine, WBC).
   • Text-Derived Indicators: Radiology reports (CT/MRI) are processed using Term Frequency–Inverse Document Frequency (TF-IDF) and a gradient boosting classifier to create a text-derived delirium probability indicator.
   • Expert Refinement: After initial Lasso-based screening, physicians manually refine feature retention to preserve clinically meaningful predictors and etiologically relevant signals.
3. Machine Teachers-Centered Evaluation (MTCE): Physicians evaluate candidate models using an interactive dashboard. Selection criteria include discrimination performance (F1-score, ROC-AUC), temporal stability (standard deviation across time segments), and calibration quality.

Dataset and Experimental Setup
The study utilizes the GEMINI (General Medicine Inpatient Initiative) dataset, comprising 3,862 labeled hospital admissions from six Toronto hospitals. The data includes administrative variables, laboratory results, medications, and radiology-derived text indicators.

• Phases: The study spans Phase 1 (April 2010–March 2015) and Phase 2 (January 2018–January 2020).
• Validation Strategy: A temporally separated holdout testing approach was used. Phase 1 data was segmented into ten 6-month intervals; models were trained on the first nine and tested on the tenth to simulate prospective prediction. A cross-phase drift test evaluated the Phase 1-trained model on Phase 2 data.
• Comparators: The proposed UC-iML-Delirium model was compared against:
  • aML-Delirium: Data-driven feature selection without physician input.
  • Baseline-Delirium: Excluded text-derived features and physician-guided refinement.
• Algorithms: Twelve classifiers were tested, including Gradient Boosting (GBC), AdaBoost, Logistic Regression, and Random Forest. GBC was selected as the top performer.
• Interpretability: Shapley Additive exPlanations (SHAP) were used to quantify feature attribution and ensure clinical interpretability.

Key Results

• Performance: The UC-iML-Delirium framework demonstrated superior overall discrimination compared to automated and baseline variants. On the prospective holdout test (Phase 1), the GBC-based UC-iML model achieved an F1 score of 0.719, Accuracy of 0.851, and ROC-AUC of 0.902.
• Temporal Robustness: The UC-iML model showed significantly stronger temporal stability. Across rolling 6-month windows, the F1 score for UC-iML-GBC was 0.701 ± 0.024, compared to 0.749 ± 0.136 for aML-Delirium and 0.402 ± 0.046 for Baseline-Delirium.
• Cross-Phase Validation: When tested on Phase 2 data (3–5 years later), the model maintained robust performance (F1 = 0.693 for GBC), though it slightly underestimated delirium prevalence, consistent with increased class imbalance in the later dataset.
• Feature Importance: SHAP analysis revealed that engineered variables, particularly a text-derived delirium probability from radiology reports, were among the strongest predictors. Other top risk factors included specific diagnosis codes (e.g., F05 for delirium in the ER), antipsychotic orders, and lactate levels.
• Clinical Trade-offs: While the UC-iML model achieved higher recall (sensitivity) and F1 scores, it showed lower precision compared to the Baseline model. The authors note that for screening purposes, minimizing missed cases (high recall) is prioritized over precision.

Significance and Claims
The paper claims that the UC-iML framework is a practical, human-in-the-loop approach that significantly improves delirium prediction compared to standard automated methods. The key contributions are:

1. Integration of Expertise: Demonstrating that physician-guided feature refinement and evaluation lead to models with better clinical relevance and temporal robustness than purely data-driven approaches.
2. Replicability: Establishing a replicable pipeline for clinical applications that balances automated efficiency with domain expert oversight.
3. Robustness: Showing that the model maintains performance stability over time and across different hospital cohorts, addressing the issue of model drift common in healthcare AI.
4. Interpretability: Using SHAP to highlight clinically meaningful signals (such as radiology text indicators), thereby fostering trust among clinicians.

The authors conclude that neither feature engineering nor physician expertise alone achieved comparable results; the synergy of both within the UC-iML framework is essential for developing reliable, clinically actionable delirium detection tools. Future work is suggested to focus on further integrating physician expertise and refining the methodology for broader clinical integration.