Enhancing Fatigue Detection through Heterogeneous Multi-Source Data Integration and Cross-Domain Modality Imputation

Technical Summary: Enhancing Fatigue Detection through Heterogeneous Multi-Source Data Integration and Cross-Domain Modality Imputation

Problem Definition
Reliable fatigue detection is critical for safety in human-machine systems (e.g., aviation, mining, transport). However, effective deployment is hindered by the "sensor accessibility" problem. High-fidelity sensors (e.g., EEG, ECG, precise pupil trackers) provide accurate data in controlled laboratory settings but are often impractical for real-world use due to cost, calibration requirements, sensitivity to environmental noise, and privacy concerns. Conversely, field-deployable sensors (e.g., PPG, HR, accelerometers) are accessible but lack the fidelity of high-end modalities.

Existing data imputation methods typically address missing data within a single domain or modality (e.g., generating more EEG from partial EEG). They fail to address cross-domain modality imputation: the challenge of estimating a modality entirely absent in a target domain (e.g., EEG in a field setting) by leveraging knowledge from a source domain where that modality exists, using only shared, deployable modalities (e.g., HR, PPG) as a bridge.

Methodology
The authors propose a heterogeneous multi-source fatigue detection framework designed to transfer knowledge from controlled-environment source domains to a sensor-constrained target domain. The core mechanism is cross-domain modality imputation.

1. Problem Formulation: The framework defines a target domain ($D_T$) with limited modalities and multiple source domains ($D_s$) with different, often richer, sensor configurations. The goal is to learn a mapping $g_s: X_{\cap} \to X_{s+}$, where $X_{\cap}$ represents shared modalities (e.g., HR, PPG, GSR) and $X_{s+}$ represents modalities present in the source but missing in the target (e.g., EEG, ECG).
2. Theoretical Basis: The approach is grounded in generalization bound analysis. Theoretical results (Theorems 1–3) suggest that incorporating label-related additional modalities tightens the generalization error bound. Furthermore, if the distributional discrepancy between the imputed modality ($\hat{a}$) and the real modality ($a$) is minimized, the target domain's performance can be improved even without direct access to the high-fidelity sensor.
3. Framework Architecture:
   • Modality Imputation: For each source domain, a regression model (specifically an MLP in the experiments) is trained to predict the missing high-fidelity modality using the shared modalities as input.
   • Domain Alignment: To reduce distribution shifts between source and target domains, the framework employs Batch Normalization and Jacobian norm regularization. These techniques stabilize feature representations and reduce the classifier's sensitivity to input perturbations.
   • Augmentation and Detection: The imputed modalities are concatenated with the original target domain features to form an augmented representation. A downstream classifier (e.g., Transformer, LSTM, CNN) is then trained on this enhanced data to predict fatigue states.
   • Training Strategy: The process is executed in stages rather than end-to-end. Imputers are first trained on source data, applied to the target data to generate synthetic features, and finally, the fatigue detector is trained on the augmented target dataset.

Key Contributions

• Problem Formalization: The paper formally defines a practical but underexplored problem setting: operating fatigue detection with accessible field sensors while leveraging knowledge from high-fidelity, deployment-constrained source modalities.
• Heterogeneous Framework: It proposes a framework that uses shared physiological modalities as a common input space to learn source-domain relationships and transfer them to estimate unavailable modalities in the target domain.
• Proof-of-Concept Validation: The study demonstrates the feasibility of this approach using a field-deployed dataset (VPFD) and two public laboratory datasets (MEFAR, FatigueSet). Crucially, the authors intentionally use well-established algorithms (MLP, LSTM, Transformer) for the core components to isolate the contribution of the framework design itself, rather than relying on novel algorithmic innovations.

Experimental Results
Experiments were conducted using the VPFD dataset (target domain, lacking EEG/ECG) augmented with data from MEFAR (providing EEG) and FatigueSet (providing EEG and ECG).

• Imputation Efficacy: Regression-based imputation (using MLP) successfully recovered missing modalities with lower Mean Squared Error (MSE) compared to other architectures like CNN1D, LSTM, and TimeMixer.
• Classification Performance:
  • Imputing missing modalities significantly improved fatigue detection accuracy compared to using only available target sensors.
  • The combination of imputed EEG and ECG yielded the most pronounced improvements. For instance, using a Transformer classifier with TimeMixer-based imputation on the augmented VPFD dataset achieved 92.86% accuracy and 97.88% ROC-AUC, compared to 82.31% accuracy and 91.84% ROC-AUC on the baseline VPFD data.
  • Ablation studies confirmed that combining Batch Normalization and Jacobian regularization provided the best stability and performance, particularly when multiple imputed modalities were present.
• Noise Sensitivity: Experiments showed that injecting random noise into missing modalities degraded performance, confirming that the gains were due to the structured information learned from source domains, not merely the addition of data dimensions.

Significance and Claims
The paper positions itself as a proof-of-concept demonstrating that knowledge transfer across differently instrumented domains is feasible. The authors claim that:

1. High-fidelity knowledge from controlled environments can be effectively transferred to real-world scenarios where those sensors are unavailable, provided shared modalities exist to bridge the domains.
2. This approach allows for the preservation of the utility of high-fidelity sensors without requiring their direct, often impractical, deployment in the field.
3. The framework successfully addresses the gap between "sensor reliability" (high in labs, low in fields) and "sensor accessibility" (low in labs, high in fields).

The authors remain modest regarding future implications, noting that the current implementation uses simple sequential enhancement and does not yet model conflict-aware weighting when sources provide inconsistent estimates. They identify future work in scaling to larger cohorts, developing adaptive multi-source weighting, and optimizing for real-time computational efficiency.