YawDD+: Frame-level Annotations for Accurate Yawn Prediction

Technical Summary: YAWDD+ – Frame-Level Annotations for Accurate Yawn Recognition on Edge Platforms

Problem Statement
Driver fatigue is a critical safety issue, responsible for approximately 24% of road crashes, with yawning serving as a key early behavioral indicator. While existing datasets like YawDD provide video-based annotations, they suffer from systematic noise due to coarse temporal labeling. In these datasets, entire video clips are labeled as "yawning" even when the majority of frames depict non-yawning behaviors (e.g., normal driving, talking, or smiling). This misalignment introduces noise that hinders the training of robust machine learning models. Furthermore, current state-of-the-art solutions often rely on cloud infrastructure or heavy video-transformer architectures that are computationally expensive and impractical for real-time, on-device deployment in vehicles where connectivity is limited and data privacy is paramount.

Methodology
To address label noise and enable efficient edge deployment, the authors propose YawDD+, a refined dataset featuring precise frame-level annotations derived from the original YawDD. The core of their approach is a semi-automated labeling pipeline incorporating human-in-the-loop verification, structured in three stages:

1. Mouth State Classification: A lightweight Convolutional Neural Network (CNN) with approximately 421,000 parameters is trained to classify mouth states as "open" or "closed." This model achieves 96% accuracy on a binary classification task, utilizing data augmentation to overcome the scarcity of existing mouth-state datasets.
2. Face Detection and Landmark Extraction: The pipeline employs YOLOv8 for robust driver face detection across varying lighting and poses. To ensure precise mouth region extraction, the system integrates MediaPipe Face Mesh to generate 468 3D facial landmarks. This approach overcomes the temporal instability of Dlib detectors in vehicular environments. The system automatically selects the driver's face (the largest bounding box) and extracts the mouth region using specific lip landmarks with configurable padding.
3. Automated Annotation with Human Verification: The trained CNN classifies the cropped mouth regions. Predictions with high confidence (>95%) are automatically accepted. Low-confidence cases are routed to a custom GUI for human verification, allowing for the correction of false positives and negatives in batches of 64 images. This process reduces manual effort by automating ~80% of the labeling while focusing human effort on challenging edge cases.

The resulting YawDD+ dataset contains 124,201 annotated "yawning" frames and 222,837 "non-yawning" frames, totaling 287,225 frames.

Key Contributions

• YawDD+ Dataset: The creation of a high-quality, frame-level annotated dataset that eliminates the systematic noise present in video-level annotations, distinguishing specific yawning instances from other mouth movements.
• Semi-Automated Pipeline: A scalable, human-in-the-loop labeling framework that significantly reduces annotation time while maintaining high data quality.
• Edge-Optimized Training and Inference: The demonstration of training and deploying models (MNasNet for classification and YOLOv11 for detection) directly on edge devices (NVIDIA Jetson NANO and AGX) without reliance on server-side computation.

Experimental Results
The authors evaluated MNasNet and YOLOv11 on the YawDD+ dataset using NVIDIA Jetson platforms:

• Accuracy Improvements: Compared to video-level supervision, training on YawDD+ improved frame accuracy by up to 6% and mean Average Precision (mAP) by 5%.
• Performance Metrics:
  • MNasNet (Classification): Achieved 99.34% accuracy and 98.30% F1-score.
  • YOLOv11 (Detection): Achieved 95.69% mAP50–95 and 98.67% F1-score.
• Efficiency:
  • Inference Latency: The models demonstrated real-time capabilities with inference times as low as 12.03 ms (classification) and 24.33 ms (detection) on Jetson AGX, translating to up to 115 FPS.
  • Training Time: On-device training was feasible, with MNasNet completing an epoch in just 8.69 minutes on Jetson AGX.
• Comparison: These results outperform prior frame-based studies (which ranged from 93.31% to 94.5% accuracy) and avoid the high inference costs and lack of edge deployment metrics associated with video-based transformer architectures (e.g., 2s-STGCN, DLS).

Significance and Claims
The paper claims that the enhanced data quality provided by YawDD+ is the primary driver for the superior performance of lightweight architectures. By removing label noise, the authors demonstrate that complex, resource-heavy video transformers are not strictly necessary for high-accuracy yawning detection. Instead, precise frame-level annotations enable efficient, lightweight models to achieve state-of-the-art accuracy while fitting within the compute and memory constraints of commodity edge hardware.

This work validates the feasibility of on-device driver fatigue monitoring systems that operate in real-time without cloud dependency, thereby addressing critical requirements for data privacy, latency, and reliability in automotive safety applications. The authors suggest that the semi-automated pipeline can be migrated to other vision tasks to reduce manual labeling burdens, and the dataset can support multimodal fatigue analysis and other domains requiring fine-grained facial behavior analysis.