Leakage-Resistant Evaluation of Calibrated Multimodal Driver Drowsiness Detection Across Drivers and Routes

Imagine you are trying to teach a robot to spot when a human is about to fall asleep at the wheel. You give the robot a camera, a smartwatch, and a dashboard sensor to watch the driver. But here's the tricky part: if you aren't careful, the robot might rely on shortcuts. It could learn to recognize the specific driver's face or the specific road they are driving on, rather than actually learning what "sleepy" looks like. It's like studying for a math test by memorizing the answers to the practice questions instead of learning the formulas. If the test questions change even a little, the robot fails. This is the problem of "data leakage" in artificial intelligence.

To build a truly safe system, we need to make sure the robot learns the universal signs of drowsiness—like heavy eyelids, slow reactions, or changes in heart rate—so it can work on any driver, on any road, even ones it has never seen before. We also need to make sure the robot doesn't just guess "awake" or "asleep" but can tell us how sure it is. If the robot is 99% sure the driver is asleep, we want to be able to trust that number. This paper dives deep into these questions, acting like a strict referee to check if previous claims about sleepy-driving detectors were actually fair or if they were just lucky guesses based on relying on shortcuts.

---

The Great Drowsiness Detective Audit

In this study, a team of researchers decided to play the role of "leakage detectives." They took two massive collections of data—one with 225,000 observations from 15 drivers on highways, rough terrain, and city streets, and another with over 87,000 skin-conductance readings from 20 people—and put them through a rigorous, no-shortcuts-allowed test. Their goal was to see if the best-performing models from previous studies could still do their job when they weren't allowed to peek at the answers or memorize the specific drivers.

The "Shortcuts" Cleanup
First, the researchers scrubbed the data clean. They realized that some of the data columns were like giving the detective the answer key. For instance, if the data included a column that literally said "Sudden Brake Happened," the computer could just guess "Drowsy" whenever a brake happened, without actually understanding the driver's state. They also removed any columns that were just different ways of writing the "sleepy" label. Once they stripped away these "shortcuts," they were left with the raw, honest signals: heart rate, grip pressure, foot movement, and vehicle data.

The Four Ways to Test
To see if the models were truly smart or just memorizers, the team used four different testing strategies, like changing the rules of a game to see who really knows the rules:

1. Random Row Split: Shuffling all the data like a deck of cards. This is the "easy mode" where the computer sees bits of the same driver and road in both its training and testing.
2. Time-Blocked: Making sure the computer only learns from the past to predict the future, like watching a movie and trying to guess the ending without seeing the last 20 minutes.
3. Driver-Held-Out: This is the "new driver" test. The computer learns from 10 drivers and is tested on 5 it has never met.
4. Route-Held-Out: The ultimate "new world" test. The computer learns on highways and rough roads but is tested on dense city traffic it has never seen before.

The Winner: The "All-in-One" Brain
The researchers tested three different types of AI brains: a simple linear thinker (Logistic Regression), a complex tree-learner called Pooled HGB (Histogram-based Gradient Boosting), and a team of specialists who voted on the answer (CQMF, or Calibrated Quality-Weighted Modality Fusion).

The surprise winner was the Pooled HGB. Think of this model as a single, super-smart detective who looks at everything at once—heart rate, foot movement, and car speed—simultaneously. It didn't try to separate the clues into different categories; it just let the data speak to each other.

• When tested on new drivers (Driver-Held-Out), this model was incredibly accurate. It got a score of 0.9990 (out of 1.0) for distinguishing awake from asleep, and a 0.9819 score for finding the rare sleepy moments.
• It was also very honest about its confidence. Its "Brier score" (a measure of how well its confidence matches reality) was a tiny 0.00355, meaning when it said "I'm 90% sure," it was almost always right.

The "Specialist Team" Failed
The researchers had hoped that the "Specialist Team" (CQMF) would be better. The idea was: have one AI watch the heart, another watch the hands, and a third watch the car, then have them vote. But this approach actually performed worse. When the team tried to combine their separate opinions, they lost the subtle connections between the signals. It turns out, a single detective who sees the whole picture is better than a committee of specialists who can't talk to each other. The "All-in-One" brain captured the fact that a specific heart rate pattern might mean something different depending on what the car is doing, something the separate specialists missed.

The "New Road" Penalty
Here is where things get a little less perfect. When the researchers tested the model on a completely new type of road (Route-Held-Out), the performance dipped.

• On highways, the model was still great.
• But on dense urban routes, the ability to spot sleepy drivers dropped. The "sensitivity" (how many sleepy drivers it actually caught) fell to 0.7216.
  This reveals a "route-transfer penalty." The model learned that sleepy drivers on highways look a certain way, but city driving is chaotic and different. If you take a model trained only on highways and drop it into a busy city, it might miss the sleepy drivers.

The "Broken Sensor" Stress Test
Real life is messy. Sensors break, wires get loose, and data gets lost. The researchers simulated this by randomly deleting 10%, 20%, 30%, and even 50% of the data to see what happened.

• The "All-in-One" model (Pooled HGB) held up surprisingly well. Even with 50% of the data missing, it still managed a decent score of 0.6630.
• The "Specialist Team" (CQMF) fell apart much faster, dropping to 0.5260 with 50% missing data.
  This suggests that having one robust model that can handle missing pieces is better than relying on a team of specialists who might all go silent if their specific sensor fails.

The Skin Conductance Mystery
The team also looked at a second dataset involving skin conductance (how sweaty your skin gets when you are stressed or sleepy). Even after removing the "shortcuts," the model still performed amazingly well, with a score of 0.9994. However, the researchers are cautious. They noted that because they didn't have the original raw video or audio logs, they couldn't be 100% sure the "sleepy" labels were perfectly accurate in the first place. It's like finding a perfect score on a test, but you don't know if the teacher graded it fairly. So, while the numbers look great, they can't claim this proves the system works in the real world yet.

The Bottom Line

This paper doesn't just tell us which AI model is the "best." It teaches us how to stop relying on shortcuts. The main takeaway is that you cannot trust a driver-drowsiness system unless you test it on drivers and roads it has never seen before.

The study found that a single, powerful model that looks at all the data together (Pooled HGB) is the most reliable. It is better than trying to combine separate models, and it is robust enough to handle missing data. However, the study also warns us that even the best model isn't perfect. If you move a system from a highway to a busy city, or if sensors start failing, the system might miss the danger.

The authors conclude that while we have a very strong tool for spotting drowsiness right now, we aren't ready to put it in every car yet. We need more real-world testing, better data from raw sensors, and a system that knows when it's confused and should ask for help. Until then, this research serves as a crucial "leakage audit," ensuring that when we do claim a system is safe, we aren't just fooling ourselves with a lucky guess.