Alert Policy and Rare-Event Metrics for Seven-Day-Ahead Agitation Forecasting in Passive Dementia Monitoring

Technical Summary: Alert Policy and Rare-Event Metrics for Seven-Day-Ahead Agitation Forecasting

Problem Statement
The paper addresses the critical challenge of forecasting agitation in dementia patients using passive home monitoring data. While digital biomarkers, specifically multi-scale entropy features derived from passive infrared sensors, show promise in predicting agitation events days in advance, a significant barrier to clinical deployment is extreme class imbalance. In the Technology Integrated Health Management (TIHM) cohort, only 4.3% of patient-days (114 out of 2,665) were labeled with agitation.

Standard evaluation metrics, such as accuracy or ROC-AUC, often obscure the practical utility of these models. At default classification thresholds (0.5), models achieve high recall (94–100%) but suffer from low precision (20–23%). This results in an unmanageable volume of false alerts, which can lead to "alert fatigue" and erode caregiver trust, rendering the system unusable for real-world passive monitoring without nurse triage.

Methodology
The study utilizes a patient-stratified split of the TIHM dataset (51 patients, 2,665 analyzable patient-days) into training (70%), validation (15%), and test (15%) sets. The binary label indicates the occurrence of agitation within the subsequent seven days.

1. Feature Engineering: The pipeline computes complexity metrics from sensor location data, including Sample Entropy, Approximate Entropy, Multi-scale Entropy (scales 1–2), and the Hurst exponent. These are aggregated into rolling 3-day and 7-day means, deviations from baseline, and slopes to capture short-term disruptions against patient-specific norms.
2. Model Architectures: Three recurrent neural network architectures were trained on 14-day histories of these normalized features:
   • LSTM (64 units with attention and dropout).
   • GRU (64 units).
   • Bidirectional LSTM with Attention (32 units per direction).
   • Training Strategy: Models utilized inverse-frequency class weights (approx. 1:4.5) to address imbalance, the Adam optimizer, and early stopping based on validation AUC.
3. Operating Point Analysis: The core methodological contribution is the evaluation of two distinct operating regimes:
   • Default Regime: A fixed threshold of 0.5.
   • Optimized Regime: Thresholds were selected on the validation set to maximize the F1-score on the precision-recall curve, then frozen for test evaluation.
4. Metrics: Beyond standard accuracy and F1, the study explicitly calculates alerts per 100 patient-weeks and Positive Predictive Value (PPV) to quantify the "alert burden" on caregivers.

Key Results
The analysis demonstrates that model architecture choice is secondary to the selection of the operating threshold in this rare-event context.

• Default Thresholds (0.5): All three models (LSTM, GRU, BiLSTM+Attention) exhibited similar performance: 94–100% recall but only 20–23% precision. This implies that under default settings, more than three out of four alerts would be false positives.
• Optimized Thresholds: When thresholds were tuned for F1-maximization:
  • LSTM: Threshold 0.6463, Precision 72.97%, Recall 56.25%.
  • GRU: Threshold 0.7060, Precision 77.14%, Recall 56.25%.
  • BiLSTM+Attention: Threshold 0.5931, Precision 81.82%, Recall 56.25%, Accuracy 91.15%, F1 0.667.
• Alert Burden Reduction: The optimized BiLSTM+Attention model reduced the expected false alarms by more than half compared to default calibration. Specifically, it generated roughly one alert per 6–7 flagged days, versus one per 2–3 days under default settings, while maintaining a precision of over 80%.

Significance and Claims
The paper argues that for rare-event forecasting in clinical settings, reporting accuracy or AUC alone is insufficient and potentially misleading. The authors claim that:

1. Threshold Selection is a Deployment Decision: The choice of operating point is a "first-class deployment decision" that dictates the clinical viability of the system. A model is not clinically actionable until its operating policy is governed to balance sensitivity with acceptable false alarm rates.
2. Actionability via Precision: Entropy-sequence models become clinically useful only when thresholds are optimized to achieve high precision (73–82%), even at the cost of reduced recall (dropping to ~56%). This trade-off is necessary to prevent caregiver overload.
3. Reporting Standards: The authors propose a reporting checklist for rare-event forecasting that must include: (i) alerts per patient per month, (ii) PPV at the deployed threshold, (iii) lead-time distribution, and (iv) policies for repeat alerts. They assert that translational relevance should be claimed based on these operational endpoints rather than leaderboard accuracy.

The study concludes that while retrospective tuning on a single validation fold has limitations, the framework of "operating-point governance" is essential for moving passive dementia monitoring from research prototypes to prospective, caregiver-actionable systems.