Validation design changes wearable stress-state classification performance in hospital nurses

Technical Summary: Validation Design Changes Wearable Stress-State Classification Performance in Hospital Nurses

Problem Statement
Wearable sensors offer continuous physiological monitoring in naturalistic environments, yet their utility for stress classification is often overstated due to flawed validation designs. A critical issue arises when dense sensor streams from a single individual generate thousands of adjacent data windows. If windows from the same person contribute to both model training and evaluation (internal window-level validation), the resulting high accuracy may reflect the model's ability to recognize specific user signatures rather than its capacity to generalize to unseen individuals. This paper investigates how validation design impacts the apparent performance of wearable-derived stress classifiers, specifically examining the discrepancy between internal window-level estimates and participant-independent (leave-one-identifier-out) evaluations.

Methodology
This study is a retrospective secondary computational benchmarking analysis of a public multimodal dataset collected from 15 female hospital nurses during the COVID-19 pandemic.

• Data Source: The analysis utilized a merged redistribution of the dataset originally reported by Hosseini et al., containing approximately 11.5 million one-second rows. The source cohort consisted of 15 nurses, though the redistributed data contained 18 unique identifier values.
• Sensors & Features: Data was captured via Empatica E4 devices, measuring electrodermal activity (EDA), heart rate (HR), skin temperature, and triaxial acceleration. Signals were segmented into 60-second windows, and 24 statistical features (mean, SD, min, max) were extracted per modality.
• Labels: The target variable was a three-level stress state (low, medium, high). These labels were derived from algorithm-proposed events that were subsequently reviewed by the nurses, rather than independent clinical diagnoses.
• Models: Six algorithms were evaluated: Random Forest, XGBoost, LightGBM, Multilayer Perceptron (MLP), Long Short-Term Memory (LSTM), and 1D Convolutional Neural Networks (CNN).
• Validation Strategy:
  1. Internal Time-Aware Window Validation: A split based on time within the merged sequence, where later windows served as the test set. This allowed windows from the same nurse to appear in both training and testing partitions.
  2. Leave-One-Subject-Out (LOSO) Validation: A participant-independent evaluation where all windows for a single identifier were held out during model fitting. This was performed only for the XGBoost model due to data availability constraints.
• Analysis Tools: SHAP (SHapley Additive exPlanations) values were used to interpret feature importance, and sensor-group ablation studies were conducted to assess the contribution of specific modalities.

Key Results

• Performance Discrepancy: Internal window-level accuracy ranged from 0.810 (MLP) to 0.990 (LSTM and 1D CNN). In contrast, when XGBoost was evaluated using LOSO validation, the mean accuracy dropped significantly to 0.661 (a decrease of 0.287). The accuracy for individual held-out identifiers varied widely, ranging from approximately 0.32 to 0.90.
• Feature Importance: SHAP analysis indicated that EDA features provided the largest contributions to the model's output, followed by skin temperature and movement features. Heart rate contributions were minimal.
• Ablation Findings: Removing EDA features reduced accuracy by 0.0606, and removing skin temperature reduced it by 0.0455. Removing heart rate or any single accelerometer axis resulted in negligible reductions (<0.009).
• Temporal Patterns: Exploratory temporal mapping revealed heterogeneous patterns across identifiers. The data did not support consistent claims that stress levels were higher during night shifts or specific time blocks (e.g., 12:00–18:00) without verified rosters.

Key Contributions

1. Demonstration of Validation Bias: The study provides empirical evidence that internal window-level validation substantially overestimates model performance for stress classification. High internal accuracy (up to 0.990) does not guarantee transportability to a new, unseen nurse.
2. Quantification of Generalization Gap: By comparing internal and LOSO results for XGBoost, the paper quantifies the significant drop in performance (from 0.948 to 0.661 mean accuracy) when moving from user-specific to user-independent evaluation.
3. Modality Dependency Analysis: The work identifies that the classifier relies heavily on EDA and skin temperature, suggesting that models trained on these specific features may not generalize well if these signals are noisy or absent in new settings.
4. Label Provenance Critique: The analysis highlights that the target labels are not independent clinical diagnoses but are partially derived from the same sensor data used for prediction, creating a potential pathway for label incorporation that inflates internal performance.

Significance and Claims
The paper concludes that the retained internal estimates substantially overstated the performance relevant to an unseen nurse. The authors assert that high internal accuracy should not be mistaken for deployment readiness.

• Operational Readiness: The findings indicate that the current models are not ready for operational use as autonomous alarms, staffing rules, or performance management tools.
• Requirements for Deployment: Before such systems can be used in practice, independent outcomes, reproducible participant-level evaluation (like LOSO), calibration, and external validation are required.
• Limitations: The authors explicitly note limitations including the small, all-female cohort from a single site, the unresolved discrepancy between 15 source nurses and 18 identifiers, the lack of independent outcomes, and the absence of confidence intervals or class-specific metrics.
• Future Direction: The paper calls for definitive evaluations that preregister intended use, collect independent time-stamped outcomes, and report uncertainty and calibration metrics, emphasizing that transportability and reproducibility must take precedence over headline window-level accuracy.