🤖 machine learning

Human Centered Non Intrusive Driver State Modeling Using Personalized Physiological Signals in Real World Automated Driving

This study demonstrates that personalized, non-intrusive driver state modeling using multimodal physiological signals and deep learning significantly outperforms generalized models (92.68% vs. 54% accuracy) in real-world automated driving, highlighting the critical need for adaptive systems that account for individual physiological variability.

Original authors: David Puertas-Ramirez, Raul Fernandez-Matellan, David Martin Gomez, Jesus G. Boticario

Published 2026-04-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: David Puertas-Ramirez, Raul Fernandez-Matellan, David Martin Gomez, Jesus G. Boticario

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a passenger in a very advanced car that can drive itself most of the time, but it still needs you to keep an eye on things and take over if something goes wrong. This is what we call "Level 2" or "Level 3" automation. The big problem? If the car gets too good at driving, you might get bored or distracted, and if the car suddenly says, "Hey, I need you to drive!" you might be too slow to react.

To solve this, cars need a "Driver Monitoring System" (DMS) to check if you are awake and paying attention. Most current systems are like a generic teacher who tries to teach every student in the class using the exact same lesson plan. They assume everyone learns and reacts the same way.

This paper argues that this "one-size-fits-all" approach is a disaster for driver safety. Instead, the researchers propose a personalized tutor that learns exactly how you specifically react.

Here is the breakdown of their study, explained with some everyday analogies:

1. The Experiment: Wearing a Smart Watch on the Highway

The researchers didn't just use a computer simulation (which is like practicing driving in a video game). They put real people in a real car on real highways.

The Gear: They gave the drivers a smart wristband (Empatica E4) that measures things like your heart rate, how sweaty your palms are (a sign of stress or focus), your body temperature, and your movement.
The Goal: To see if these physical signals can tell the car if the driver is "zoning out" or "fully alert."

2. The Magic Trick: Turning Heartbeats into Pictures

Computers are really good at recognizing faces in photos, but they are bad at looking at a list of numbers (like a heart rate chart).

The Analogy: Imagine trying to recognize a song just by reading a list of notes. It's hard. But if you turn that list of notes into a visual pattern (like a squiggly line drawing), a computer can "see" the shape of the song instantly.
The Method: The researchers took the raw data from the wristband and turned it into 2D images. They then fed these images into a famous AI brain (called ResNet50) that was originally trained to recognize cats and dogs. Surprisingly, this AI became an expert at recognizing "driver states" just by looking at these heartbeat pictures.

3. The Big Discovery: You Are Not Your Neighbor

This is the most important part of the paper. They tested two theories:

Theory A (The Generalist): Train one AI on data from four different people, and use that one AI to monitor all four.
Theory B (The Specialist): Train a separate AI for each person, using only their data.

The Results were shocking:

The Generalist (One model for all): It was terrible. It got the driver's state right only about 54% of the time. That's barely better than flipping a coin!
The Specialist (One model per person): It was amazing. It got the state right about 93% of the time.

Why?
Think of it like fingerprints.

When you are stressed, your heart might race. But for Person A, stress might make their heart race fast and their palms sweat a lot. For Person B, stress might make their heart beat slowly and their skin get cold.
A "Generalist" AI tries to find a "universal stress pattern." It gets confused because Person A and Person B look completely different to it.
A "Specialist" AI learns your specific fingerprint. It knows, "Oh, when this driver's heart slows down slightly, they are actually getting drowsy," whereas for another driver, that same signal means nothing.

4. Why This Matters for the Future

The paper concludes that for self-driving cars to be safe, they can't just use a generic "driver monitor." They need to be adaptive.

The Old Way: The car assumes all humans are the same.
The New Way: The car should start with a basic understanding, but as you drive with it, it should learn your unique biological rhythm. It should say, "I know how you look when you are tired, so I will ask you to take over before you actually fall asleep."

Summary

This study proves that human bodies are too unique for generic computer models. Just as you wouldn't wear a suit that fits everyone in the world, a safety system for a self-driving car shouldn't try to fit every driver into the same box. To keep us safe, the car needs to get to know you personally, learning your specific heartbeat and stress signals to know exactly when you need to take the wheel.

1. Problem Statement

In vehicles with SAE Levels 2 and 3 automation (Partial and Conditional Driving Automation), the driver remains responsible for supervising the system and responding to Take-Over Requests (TORs). A critical challenge is the "irony of automation," where high system reliability leads to reduced driver attention.

Current Limitations: Existing Driver Monitoring Systems (DMS) largely rely on generalized, user-agnostic models (often based on computer vision) that ignore significant inter-individual physiological variability.
Research Gaps: Most prior physiological studies rely on controlled simulations, which fail to capture the stochastic nature and psychological pressure of real-world driving. Furthermore, there is a lack of empirical evidence regarding whether personalized models using non-intrusive sensors outperform generalized models in real-world scenarios.

2. Methodology

Experimental Setup

Vehicle: A Toyota Prius PHEV equipped with Comma OpenPilot (SAE Level 2), where the system controls steering and speed, but the driver must supervise.
Environment: Real-world public highways. A human co-pilot monitored safety and enacted 16 distinct scenarios (e.g., lane changes, induced inattention, unexpected events).
Sensors: Empatica E4 wristbands (non-intrusive) capturing seven data streams:
- Electrodermal Activity (EDA)
- Heart Rate (HR)
- Skin Temperature (TEMP)
- Blood Volume Pulse (BVP)
- 3-axis Accelerometer (ACCx, ACCy, ACCz)
Participants: 4 male volunteers (3 aged 25–29, 1 aged 45–49) with prior OpenPilot experience.

Data Processing: Signal-to-Image Transformation

To leverage Convolutional Neural Networks (CNNs) designed for images, the authors transformed 1D physiological time-series into 2D image representations.

Preprocessing: All signals were resampled to a common frequency of 4 Hz.
Windowing: A 30-second window with a 3-second step size was used.
Normalization: A "Combined Metrics" approach (balancing global and local normalization) was selected to preserve contrast and scale.
Encoding Techniques Evaluated:
1. Recurrence Plots (RP): Binary and Continuous.
2. Gramian Angular Fields (GAF): Summation (GASF) and Difference (GADF).
3. Markov Transition Fields (MTF): With 4 and 128 states.
Result: Continuous Recurrence Plots (RP) yielded the highest accuracy and were selected for the final model.

Model Architecture

A multimodal deep learning architecture was proposed:

Feature Extraction: Seven input images (one per sensor signal) are processed independently using pre-trained ResNet50 (transfer learning). The penultimate layer extracts a 2048-dimensional vector per signal.
Fusion: The seven vectors are concatenated into a single vector of size 14,336.
Dimensionality Reduction: Principal Component Analysis (PCA) reduces the vector to 100 components (optimized for convergence vs. overfitting).
Classification: A fully connected neural network (2 FC layers: 100→25→10) with ReLU activation, followed by a Softmax layer outputting 4 classes:
- Low-Low (LL), Low (L), High (H), High-High (HH).
- These classes represent granular levels of "Fallback Readiness" based on alarm management standards.

3. Key Contributions

Real-World Physiological Sensing: The first study to utilize non-intrusive wearable sensors for driver state modeling in a real-world SAE Level 2 environment, moving beyond simulation.
Signal-to-Image Pipeline: A novel pipeline converting multimodal physiological time-series into 2D images, enabling the use of powerful pre-trained CNNs (ResNet50) for driver state classification.
Empirical Evidence of Variability: Demonstrated that physiological patterns of driver awareness exhibit massive inter-individual variability, rendering generalized models ineffective.
Personalization Validation: Provided strong evidence that personalized models significantly outperform generalized (combined-user) models, advocating for adaptive, human-centered DMS.

4. Results

Performance of Personalized Models

Accuracy: Personalized models (trained and tested on the same user) achieved an average accuracy of 92.68% (ranging from 87.52% to 96.52% across users).
Stability: Low standard deviation (±0.6% to ±1.6%) indicates robust performance.

Performance of Generalized/Combined Models

Cross-User Testing: When a model trained on one user was tested on another, accuracy dropped drastically to 40.62% – 63.52%.
Leave-One-Out Testing: Models trained on 3 users and tested on the 4th (unseen) user achieved only 45.23% – 57.55% accuracy.
Data Mixing Penalty: Even when a user's data was included in the training set of a combined model, accuracy dropped by 5% to 12% compared to a purely personalized model trained only on that user's data.

Comparative Analysis

Overall Gap: Personalized models averaged 92.68% accuracy, while combined (generalized) models averaged 53.76%.
Statistical Significance: The difference is statistically significant ( $p < 0.001$ ) across Accuracy, Precision, Recall, and F1-Score.
Recall Instability: Generalized models showed erratic behavior, with recall for specific states varying wildly between drivers (e.g., 96% recall for User 3 vs. 49% for User 2 on the same state), making them unreliable for safety-critical applications.

5. Significance and Implications

Necessity of Personalization: The study proves that a "one-size-fits-all" approach fails for physiological driver monitoring due to high individual variability. Future systems must be adaptive.
Hybrid System Design: The authors propose a hybrid architecture: deploy a generalized model for immediate baseline monitoring, then progressively collect data to train a personalized model for the specific driver.
Safety Criticality: Generalized models risk missing critical states (low recall) or generating false alarms, which could compromise the safety of human-automation collaboration.
Methodological Shift: Validates the use of signal-to-image encoding with transfer learning as a viable, high-precision method for processing physiological data in real-world settings.

Conclusion

The paper concludes that while generalized models offer immediate deployment, they lack the reliability required for safety-critical driver monitoring. Personalized modeling is essential for accurate driver state assessment in automated vehicles. Future work should focus on transfer learning frameworks that can bridge the gap between immediate availability and long-term personalized accuracy.