Technical Summary: Predicting Cognitive Load in Immersive Driving Scenarios with a Hybrid CNN-RNN Model

1. Problem Statement

The paper addresses the critical challenge of predicting driver cognitive load (CL) under high-stress, real-world driving conditions. While existing literature extensively utilizes physiological signals to assess cognitive load, most studies focus on moderate or low load levels within simplified environments (e.g., monotonous highways without traffic). Furthermore, prior research often relies on binary classification (e.g., fatigued vs. alert) and neglects complex environmental factors such as nighttime driving and adverse weather (rain).

This study specifically targets the prediction of three distinct levels of cognitive load (low, moderate, high) induced by an auditory n-back task while drivers operate in a driving simulator under low visibility conditions (nighttime and rainy weather). The research aims to overcome the limitations of previous models by integrating multi-modal data sources—functional near-infrared spectroscopy (fNIRS), eye-tracking, and driving behavior metrics—within a realistic, traffic-filled simulation environment.

2. Methodology

2.1 Data Collection and Experimental Setup

• Participants: 10 healthy adults (9 males, 1 female).
• Environment: A driving simulator ("Next Level Racing Motion Platform V3") using Euro Truck Simulator 2 to simulate nighttime and stormy weather on a freeway (100 km/hr speed limit).
• Cognitive Task: An auditory version of the n-back task with three difficulty levels (0-back, 1-back, and 2-back) served as the secondary cognitive load inducer.
• Data Modalities:
  • Physiological: fNIRS (OBELAB device, 204 channels) and eye-tracking (Pupil Core glasses).
  • Driving Behavior: Vehicle dynamics including speed, angular velocity (X, Y, Z), linear acceleration (X, Y, Z), steering wheel angle, throttle response, and braking response (recorded via ETS2 SDK).

2.2 Data Preprocessing

• Normalization: Standard scalar functions were applied to normalize all features to a mean of zero and standard deviation of one.
• Synchronization: Driving simulator data (high frequency) was downsampled to match the sampling rate of fNIRS and eye-tracking signals to ensure temporal alignment.
• Feature Selection: The authors employed the Extra Trees Classifier to compute feature importance (Gini importance) and select the optimal subset of features. This was compared against traditional methods: Principal Component Analysis (PCA), Analysis of Variance (ANOVA), and Variance Threshold.
  • fNIRS Selection: The top 20 channels were selected based on Extra Trees importance, revealing that oxygenation channels ("O2") were more prevalent than deoxygenation channels ("R").
  • Driving Behavior Selection: Car speed and throttle response were identified as the most significant features, whereas braking response showed lower importance compared to previous ANOVA-based findings.

2.3 Proposed Architecture: Hybrid CNN-RNN

The study proposes a neural network architecture combining 1D Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) to capture both spatial and temporal dependencies in the data.

• Input: Normalized, multi-modal data (fNIRS, eye-tracking, driving behavior).
• Stage 1 (Spatial Feature Extraction): Two 1D Convolutional layers with kernel sizes of 3 and output channels of 16 and 32, respectively, followed by ReLU activation functions.
• Stage 2 (Temporal Modeling): An RNN layer (configured with 2 layers, input size 32, hidden size 64) to capture sequential dependencies and temporal evolution of cognitive load.
• Output Layer: The RNN output is flattened, passed through a fully connected layer (size 64), and processed by a sigmoid activation function for classification.
• Training: The model was trained using the Adam optimizer (learning rate 0.001) and cross-entropy loss function over 1000 epochs with a 70/30 train/validation split.

3. Key Contributions

1. Novel Data Combination: This is the first study, to the authors' knowledge, to combine fNIRS, eye-tracking, and driving behavior data specifically for predicting three levels of cognitive load in low-visibility, adverse weather conditions.
2. Advanced Feature Selection: The application of the Extra Trees classification method for feature selection demonstrated superior performance in identifying relevant features compared to ANOVA, PCA, and Variance Threshold methods.
3. Hybrid Model Efficiency: The proposed CNN-RNN architecture achieves high accuracy with fewer parameters than the authors' previous Conv-LSTM model, demonstrating improved efficiency in capturing complex spatiotemporal patterns.
4. Realistic Scenario Focus: Unlike many prior studies using monotonous highway scenarios, this work validates models under challenging conditions (night/rain) with active traffic simulation.

4. Results

The proposed model was evaluated against various feature extraction methods and a previous CNN-LSTM baseline.

• Feature Selection Performance:

  • Extra Trees yielded the highest performance: 99.91% Accuracy, 0.9991 F1-score, and 0.9999 AUC.
  • ANOVA achieved 99.60% Accuracy.
  • PCA and Variance Threshold performed significantly lower (approx. 90.6% and 97.8% respectively).
  • Confusion matrices confirmed that the Extra Trees method resulted in negligible misclassification compared to other methods.

• Model Comparison (CNN-RNN vs. CNN-LSTM):

  • Combined Data (Physiological + Driving): The CNN-RNN model improved accuracy from 99.82% (CNN-LSTM) to 99.99%.
  • Driving Behavior Data Only: The CNN-RNN model showed a significant improvement, increasing accuracy from 87.26% to 92.02%.

• Feature Importance Insights:

  • fNIRS analysis indicated that oxygenation levels are more critical for CL prediction than deoxygenation levels.
  • Driving behavior analysis highlighted car speed and throttle response as primary indicators, with braking showing less correlation under the Extra Trees method compared to previous ANOVA findings.

5. Significance and Claims

The paper claims that the integration of CNNs for spatial feature extraction and RNNs for temporal sequence modeling, combined with Extra Trees for feature selection, provides a robust framework for predicting cognitive load. The study highlights that:

• The proposed architecture is more efficient (fewer parameters) and accurate than previous Conv-LSTM approaches.
• The Extra Trees method effectively reduces dimensionality while retaining the most informative features, preventing overfitting in high-dimensional fNIRS data.
• The model remains effective even when relying solely on driving behavior data, suggesting potential for systems that do not require invasive physiological sensors.

The authors conclude that this approach enhances the reliability of cognitive load assessment in complex, real-world driving scenarios, offering a foundation for advanced Driver Assistance Systems (ADAS).

6. Limitations and Future Directions (as stated by authors)

The paper maintains a modest scope regarding its claims and explicitly notes the following limitations and future directions:

• Sample Size: The study involved a small cohort (10 participants), necessitating validation with larger sample sizes.
• Task Scope: The research focused only on the cognitive load imposed by the n-back secondary task, not the full spectrum of external demands drivers face.
• Difficulty Levels: Only three difficulty levels were examined.
• Future Work: The authors suggest future research should focus on:
  • Validating findings with larger datasets and diverse driving conditions.
  • Exploring prediction using vehicle dynamics alone.
  • Utilizing higher degree-of-freedom motion simulators with advanced motion cueing.
  • Investigating different types of secondary tasks to understand varied physiological and vehicle dynamic responses.