Technical Summary: Feature Importance Analysis for Patient Management Decisions

Problem Statement

The emergence of electronic health records (EHRs) has enabled the creation of complex clinical datasets characterized as multivariate time series. These datasets contain sequences of clinical findings, lab values, measurements, and medication actions. However, a significant challenge remains in analyzing these datasets: the complexity of the time series and the vast number of possible temporal features that can be generated to characterize a patient's state.

The core problem addressed in this paper is identifying which specific temporal characteristics and features of clinical data most significantly influence physician decision-making regarding the ordering of laboratory tests and the prescribing of medications. Key questions include whether recent patient history is more critical than distant history, whether absolute values or trends are more predictive, and whether physicians rely on simple temporal constraints or complex interactions between variables.

Methodology

Dataset

The analysis utilizes the Post-surgical Cardiac Patient (PCP) database, comprising de-identified records of 4,486 post-surgical cardiac patients treated at a University of Pittsburgh Medical Center (UPMC) teaching hospital. The data, sourced from the MARS system, includes demographics, progress notes, all lab and test results (standard and special), medication databases, microbiology, EKG, radiology, and financial charges.

Data Segmentation:
To facilitate analysis, the continuous time-series data was converted into a vector space representation of discrete patient states. Each patient record was segmented into 24-hour intervals, creating a snapshot of the patient state at 8:00 AM each day. Only information available up to the segmentation point was used. This process generated 30,828 patient state examples. Each state was linked to binary decision outcomes (True/False) for the subsequent 24 hours:

• Lab Order Decisions: Whether a specific lab was ordered.
• Medication Decisions: Whether a specific medication was administered.

Feature Engineering

The study generated 9,223 distinct features across five categories:

1. Laboratory Tests (LABs): Features included last value, second-to-last value, first value, time since last order, pending status, and trend indicators. For continuous/ordinal labs, features included differences between values, slopes, percentage changes, and comparisons to nadir (minimum) and horizon (maximum) values.
2. Medications (MEDs): Features included current status, time since last administration, time since first administration, and time since the last status change.
3. Visit/Demographics: Static features including age, sex, and race.
4. Procedures: Features capturing 36 different procedures (e.g., heart valve repair), including time since last/first occurrence, occurrence within the last 24 hours, and historical occurrence.
5. Heart Support Devices: Binary indicators for the current use of ECMO, Balloon counter pulsation, Pacemakers, or other assist devices.

Analytical Approach

The authors employed two primary analytical methods:

1. Univariate AUC Analysis:

   • The significance of individual features was assessed using the Area Under the Receiver Operating Characteristic (AUC) score.
   • Features were grouped into 40 temporal categories (e.g., "Time since last LAB") and 5 broad categories (Labs, Meds, Procedures, Demographics, Devices).
   • Importance was measured by counting how frequently a feature within a category achieved the highest AUC for predicting a specific decision.

2. Multivariate Predictive Analysis:

   • To evaluate the benefit of combining features, a linear Support Vector Machine (SVM) classifier was trained.
   • Models were trained on a subset of 2,900 patients using the top 1, top 3, and top 30 features based on univariate AUC scores.
   • Performance was evaluated on the remaining patient cases (independent test set) to determine if multivariate models significantly outperformed single-feature models.

Key Results

Prediction of Lab Orders

• Dominant Feature Categories: Lab order decisions are primarily driven by features derived from previous lab results and procedures.
• Temporal Patterns: The most influential temporal features are "time since last lab order" and "time since last lab result." This suggests that routine scheduling and the passage of time since the last test are strong predictors.
• Values vs. Trends: Absolute values (e.g., last value, nadir, horizon) were found to be more significant than trend features. This indicates that decision-making is often driven by value-based patterns, with trends serving only as refinements.
• Cross-Lab Dependencies: When predicting the order of a different lab, the "last value" of the current lab is the most significant predictor, likely due to abnormal values triggering follow-up tests or the ordering of test panels.

Prediction of Medication Orders

• Dominant Feature Categories: Medication orders (specifically new commissions) are predominantly predicted by procedure features.
• Temporal Patterns: The "time since last procedure" is often the most predictive feature. For example, the administration of Papaverine is highly predictable (AUC = 95%) based on the time elapsed since a Coronary Artery Bypass.
• Medication Interactions: The presence of one medication is a strong predictor for the presence or absence of another, reflecting complementary administration or contraindications due to drug interactions.

Multivariate Performance

• Single Feature Sufficiency: The multivariate analysis revealed that adding more features (from top 1 to top 30) often yields diminishing returns or even slight decreases in performance for certain variables (e.g., GLU lab).
• Interpretation: This suggests that for many clinical decisions, a single, simple temporal feature is often sufficient to capture the prevailing decision pattern. Complex multivariate models do not necessarily provide a significant advantage over simple patterns in this context.

Significance and Claims

The paper claims that the complexity of multivariate time series in clinical data can often be reduced to relatively simple temporal characterizations without sacrificing predictive accuracy. Specifically:

1. Simplicity of Decision Patterns: Physician decisions regarding lab orders and medications are frequently predicted well by simple patterns, such as the last value of a single test result or the time elapsed since a specific event (procedure or previous order).
2. Value over Trend: Absolute values and time constraints appear to be more critical to decision-making than complex trend analysis or multi-variable interactions.
3. Feature Selection Efficiency: The study demonstrates that identifying a few key features is often sufficient for prediction, implying that extensive feature engineering or complex models may not be necessary for understanding or predicting these specific clinical management decisions.

The authors conclude that their analysis provides a foundation for understanding the "what" and "how" of clinical decision-making, suggesting that future work should focus on expanding the analysis of feature dependencies and developing advanced feature selection techniques for more complex multivariate models.