Conditional outlier detection for clinical alerting

Problem Statement
Medical errors remain a persistent and serious issue in healthcare practice. While current computer tools for patient monitoring exist, they are primarily knowledge-based, relying on rules extracted a priori from clinical experts. These systems are time-consuming to construct and often suffer from incomplete clinical coverage. The authors propose a data-driven alternative: a monitoring and alerting framework that utilizes stored clinical information from past patient cases to identify statistical outliers (anomalies) in patient-management actions. The core hypothesis is that patient-management actions that are unusual with respect to past patients and their specific conditions may indicate potential medical errors, warranting a clinical alert.

Methodology
The paper presents a framework for conditional anomaly detection, which identifies patient-management actions that are highly unusual given a specific patient's condition and historical data. The methodology proceeds through three primary steps:

1. Segmentation and Transformation:

   • Complex multivariate time-series data from Electronic Health Records (EHRs)—including lab tests, medications, procedures, and diagnoses—are segmented at discrete time points (e.g., every 24 hours).
   • Data observed up to a segmentation point is transformed into a vector space representation of the patient state ($x$). This representation uses a fixed set of features to summarize temporal information, including:
     • Lab Features: For categorical results, features include last/second-to-last values and time since last value. For continuous/ordinal values, features include slopes, percentage changes, nadir (minimum), and apex (maximum) values.
     • Medication Features: Indicators of current usage, time since first administration, time since last administration, and time since last order change.
     • Procedure Features: Indicators of whether a procedure was performed, time since first performance, and time since last performance.
   • Patient-management actions ($y$) occurring between segmentation points are linked to the preceding patient state. Actions are binary (true/false), representing whether a specific lab was ordered or a medication was administered.

2. Learning Predictive Models:

   • The system learns a predictive model for the conditional distribution $P(y | x)$ for specific management actions (lab orders and medication orders/continuations).
   • Support Vector Machines (SVM) are used as the discriminative learning model to separate classes of actions.
   • To handle the high dimensionality of features (thousands of potential variables), a greedy feature selection process is employed. Features are grouped by category (e.g., all hemoglobin features). Groups are evaluated via internal cross-validation (using AUC) and incrementally added to the model if they improve predictive performance.
   • Probabilistic outputs are derived from the SVM decision boundary using Platt's scaling method to estimate $P(y=1 | x)$.

3. Anomaly Detection and Alerting:

   • Anomaly Score: The degree of anomaly for an action $y$ given state $x$ is calculated as $Anom(x, y) = 1 - P(y | x)$. A low probability of the observed action implies a high anomaly score.
   • Alert Score: To address the issue that patient states change over time, the system assesses the anomaly of an action $y_{t-1}$ against both the patient state at the time of the action ($x_{t-1}$) and the subsequent state ($x_t$). The alert score is defined as the minimum of these two anomaly scores: $Alert(x_t, y_{t-1}) = \min(Anom(x_{t-1}, y_{t-1}), Anom(x_t, y_{t-1}))$. An alert is raised if this score exceeds a threshold, ensuring the deviation is persistent across consecutive time steps.

Evaluation and Results
The framework was evaluated using retrospective data from 4,486 post-cardiac surgical patients at a large teaching hospital in Pittsburgh.

• Data Split: Data was divided into a training set (pre-2005) and a test set (2005 and later). The segmentation process yielded 51,492 patient-state instances (30,828 for training, 20,664 for testing).
• Model Selection: Only models with an Area Under the Curve (AUC) of 0.68 or higher were utilized.
• Expert Review: From the test set, 222 alerts were selected for evaluation by a panel of 15 critical care experts (fellows and faculty). The selection criteria included a minimum anomaly score of 0.15 and a focus on the strongest anomalies.
• Findings:
  • Using a majority rule (2 out of 3 reviewers), 121 of the 222 alerts (54.5%) were deemed clinically useful.
  • There is a positive correlation between the alert score and the true alert rate. Alerts with low scores had a true useful rate of 19%, while those with high scores reached 72%.
  • The ROC analysis yielded an AUC of 0.64, which is statistically significantly different from 0.5 (random chance), confirming that higher alert scores are indicative of higher true alert rates.

Key Contributions and Significance
The paper claims to present the most comprehensive evaluation of their conditional outlier methodology to date. Its primary contributions include:

• Data-Driven Approach: Demonstrating that anomaly-based alerting can serve as a complement to, or standalone system for, knowledge-based alerting, particularly in clinical sub-areas where expert rules are not yet available.
• Conditional Framework: Moving beyond simple outlier detection to a conditional approach where actions are evaluated relative to specific patient states and historical data.
• Clinical Validation: Providing empirical evidence that statistical anomalies in patient management are positively correlated with clinically meaningful alerts, supporting the hypothesis that this approach can help identify medical errors.
• Scalability: Successfully building and testing models on thousands of patient features and hundreds of actions extracted from real-world EHRs.

The authors conclude that while not all statistical outliers correspond to errors, the method is "worthwhile" for raising alerts due to its ability to flag errors often enough to be clinically useful, with reasonably low false alert rates. Future work is planned to refine temporal features, explore different time discretizations, and evaluate the system in additional clinical domains.