Investigating Driver-Specific Variability in Vision-Based Driver Fatigue Monitoring: A Subgroup Evaluation Design Tested in a Controlled Simulation

Technical Summary: Investigating Driver-Specific Variability in Vision-Based Driver Fatigue Monitoring

Problem Statement
Vision-based driver fatigue monitoring systems are typically validated on pooled datasets where training and test sets contain overlapping or similar drivers. This approach yields a single aggregate accuracy metric that obscures significant performance disparities across different driver subgroups. Drivers vary in eye geometry, eyewear usage (spectacles vs. sunglasses), head pose, and driving illumination conditions. A rule that performs well on average may fail for specific groups (e.g., narrow-eyed drivers or those wearing sunglasses). Existing literature often treats these factors as secondary or aggregates results by algorithm rather than by driver characteristics, leaving the generalizability of fatigue detection systems unverified across diverse populations.

Methodology
The paper proposes and tests a subject-disjoint, per-subgroup evaluation design using a controlled generative simulation rather than real-world data. The study does not estimate the accuracy of systems on real drivers but aims to expose mechanisms and define analysis protocols for future real-data studies.

• Simulation Design: A generative model simulates 240 drivers per replicate (30 replicates total), each with specific factor levels:
  • Eye Geometry: Defined by open-eye Eye Aspect Ratio (EAR) levels (narrow, medium, wide).
  • Eyewear: None, spectacles, or sunglasses (modeled as a "compression" of the EAR signal toward a generic prior).
  • Head Pose: Frontal, moderate, or large (affecting noise and frame drop rates).
  • Illumination: Normal, low, or night (affecting noise and frame drop rates).
  • Physiology: Simulated blink rates and drowsy states (slow blinks, partial lid droop) generate synthetic EAR traces at 15 fps.
• Classifiers: Two detection rules are compared:
  • C0 (Fixed Threshold): A universal threshold ($m < 0.20$) for all drivers.
  • C1 (Per-Driver Calibrated): Estimates a driver's specific open-eye level ($\hat{E}_{open}$) from 60s of alert driving and sets a dynamic threshold ($m < 0.6 \cdot \hat{E}_{open}$).
• Evaluation Protocol:
  • Subject-Disjoint Splits: 50% of drivers for training (calibration), 50% for testing. No driver appears in both sets.
  • Metrics: Accuracy, Recall, and False Positive Rate (FPR) are calculated per driver and aggregated by subgroup.
  • Statistical Checks: Kruskal–Wallis tests assess subgroup differences; Wilcoxon signed-rank tests compare C0 vs. C1.
  • Robustness: A null-model check ($s=0$, no factor effects) and an effect-size sweep ($s=0$ to $2$) validate the sensitivity of the analysis. An independent re-implementation confirms the results.

Key Contributions

1. Evaluation Design: A reusable framework featuring four annotated subgroup factors, subject-disjoint splits, per-subgroup metrics, and a null-model check.
2. Generative Simulation: A documented simulation that isolates specific driver variability factors to test detection mechanisms without the noise of real-world data collection.
3. Quantitative Findings on Variability:
   • Eye Geometry: Significantly impacts fixed-threshold rules (C0), causing high false positives for narrow-eyed drivers and low recall for wide-eyed drivers. Calibration (C1) largely eliminates this imbalance.
   • Eyewear: Sunglasses severely degrade the performance of calibrated rules (C1). The compression of the signal by sunglasses causes the calibrated threshold to fall below the actual closed-eye signal, leading to missed detections. This is identified as a design-parameter issue (specifically the closure fraction $\alpha$) rather than an inherent failure of calibration.
   • Illumination: Night conditions increase false positives for fixed rules (C0) but are better handled by calibrated rules (C1), though the mechanism involves noise interactions with the model.
   • Head Pose: Had minimal effect at the assumed magnitudes in the simulation.
4. Sample Size Guidance: Power analysis indicates that detecting large effects (e.g., eyewear impact on C1) requires ~8 drivers per subgroup, moderate effects ~16–20, and small effects ~30–40. Current public datasets (14–60 subjects) are often insufficient for robust subgroup inference.
5. Reporting Checklist & Tools: A dashboard (Streamlit/Plotly) and a checklist for subgroup reporting, emphasizing subject-level bootstrapping and the separation of calibration data.

Results

• Overall Performance: In the simulation, the fixed rule (C0) achieved higher mean accuracy (0.81) than the calibrated rule (C1, 0.78), primarily because C1 struggled with sunglasses.
• Subgroup Disparities:
  • C0: Narrow-eyed drivers suffered a high FPR (0.36) compared to medium (0.07) and wide (0.04) eyes.
  • C1: Sunglasses drivers saw a dramatic drop in accuracy (0.62) and recall (0.24) compared to those without eyewear (0.80).
• Calibration Efficacy: Calibration reduced disparities caused by eye geometry and night lighting but introduced a new vulnerability to sunglasses. The study suggests that a "reliability gate" (e.g., detecting signal compression) is necessary to mitigate the sunglasses failure mode.
• Independence: The independent re-implementation confirmed the qualitative patterns, with quantitative differences within 0.05, validating that the findings stem from the model structure.

Significance and Claims
The paper explicitly states that its findings are conditional on the simulation's assumptions and do not constitute evidence regarding real drivers. The significance lies in:

• Defining Hypotheses: Establishing testable hypotheses about how specific driver factors (geometry, eyewear, pose, light) affect fatigue detection.
• Protocol Specification: Providing a rigorous reporting checklist and sample-size requirements for future real-data studies to ensure they can detect subgroup disparities.
• System Design Guidance: Suggesting that practical systems should employ per-driver calibration combined with a signal-quality gate (to handle sunglasses) and report performance per subgroup rather than as a single pooled metric.
• Resource Assessment: Highlighting that existing public datasets are often too small or lack the necessary annotations (e.g., standardized skin tone or head pose) to support robust subgroup analysis without pooling across datasets.

The author concludes that while the simulation shows calibration can shift error types (fixing geometry issues but exacerbating eyewear issues), the ultimate value of these findings must be determined by a planned real-data study using the proposed evaluation design.