💻 computer science

Multimodal Driver Visual Load Assessment and Prediction under Dynamic Lighting Environments Using RaceGrid-OPTICS and WA-PSRSMformer

This study proposes a multimodal framework that integrates eye-movement, illumination, and vehicle-speed data with a novel RaceGrid-OPTICS clustering algorithm and WA-PSRSMformer deep learning model to accurately assess and predict driver visual load under dynamic lighting conditions, achieving 90.13% accuracy in tunnel environments.

Original authors: Noman Iqbal, Yishui Zhu, Fenjiao Wang, Luyang Wang

Published 2026-09-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Noman Iqbal, Yishui Zhu, Fenjiao Wang, Luyang Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Driving is an act of constant negotiation between the human mind and the road. For most of us, this negotiation happens without conscious thought, but it relies heavily on one sense: sight. When the light changes suddenly—stepping from a bright, sun-drenched highway into the dim interior of a tunnel, or driving through a stretch of road where tree shadows flicker rapidly across the windshield—the eyes and brain must work overtime to adjust. This sudden demand for visual information creates a state known as visual load. It is the mental effort required to process what the eyes see. When this load becomes too heavy, a driver's ability to notice hazards, recognize shapes, and react to dangers can slip, turning a routine commute into a potential risk. While we know that these lighting shifts are dangerous, understanding exactly how much mental strain they place on a driver in real time has remained difficult to measure.

A team of researchers at Chang'an University has developed a new way to peer into this invisible mental effort. They set out to build a system that can not only measure a driver's visual load as it happens but also predict when it is about to spike, all while the car is moving through complex, changing light. To do this, they moved beyond the controlled, artificial environments of driving simulators and took their experiment onto real roads. They recruited ten drivers and equipped a standard sedan with a suite of sensors: a wearable camera to track where the driver was looking and how their pupils reacted, a dashboard camera to record the road ahead, and a light meter to measure the exact brightness of the environment. The drivers navigated three distinct scenarios: a long, straight road with steady light, a shaded section where sunlight flickered through trees, and a tunnel where the light shifted dramatically from bright daylight to darkness and back again.

The researchers collected a massive amount of data, syncing the drivers' eye movements with the changing light and the car's speed. They focused on specific biological signals, such as the size of the pupils and how quickly they changed, along with where the eyes were fixed and how often they jumped to new points. These signals are the body's honest reaction to the brain's workload; when the eyes struggle to adapt to a sudden change in light, the pupils and gaze patterns shift in predictable ways. However, turning this raw stream of numbers into a clear picture of "high load" or "low load" is a complex puzzle. The team first used a sophisticated method to group similar patterns together, effectively teaching a computer to recognize the difference between a relaxed drive and a stressful one without relying on pre-set rules. They found that this approach could successfully sort the data into five distinct levels of visual load, from calm to highly strained.

Once the system could identify these states, the researchers turned to the harder task of prediction. They wanted to know if the system could look at the current data and guess what the driver's visual load would be a few moments later. They tested several different computer models, including some that are good at spotting patterns in images and others that are designed to remember sequences of events over time. The most successful approach was a type of artificial intelligence model known as a Transformer, which is particularly good at understanding how one event leads to another in a long sequence. To make this model even better for the specific challenge of driving, the team added three key improvements. First, they taught it to pay extra attention to the rare, high-stress moments, which happen less often than calm driving but are the most critical to catch. Second, they streamlined its internal processing so it could handle long stretches of driving data without getting bogged down by unnecessary calculations. Finally, they optimized its memory usage, allowing it to run efficiently on hardware that might be found in a real car.

The results of this real-world testing were striking. In the most challenging environment—the tunnel, where light shifts abruptly—the new system predicted the driver's visual load with an accuracy of over 90 percent. This was significantly better than the other models they tested, which struggled to keep up with the rapid changes. The system was also remarkably efficient, processing the data in a fraction of the time required by the older models, a crucial factor for any technology that needs to work instantly in a moving vehicle. The study confirmed that the combination of eye-tracking data, light measurements, and vehicle speed provides a complete picture of driver stress. It showed that while straight roads are generally easy to predict, the moments entering and exiting a tunnel create the most complex visual demands, often pushing drivers into high-load states that the system could detect and anticipate.

This work does not just offer a way to measure stress; it offers a blueprint for a safer future. By understanding exactly how lighting conditions strain a driver's vision, engineers can design better warning systems that alert drivers before they become overwhelmed. The system proved that it is possible to monitor the invisible mental workload of driving in real time, using the natural reactions of the human eye. As roads become more complex and autonomous driving systems take on more responsibility, having a reliable way to gauge the human element's capacity to see and react will be essential. The researchers have shown that with the right tools, we can see the unseen burden of the road and perhaps, in doing so, prevent the accidents that happen when that burden becomes too heavy to carry.

Technical Summary: Multimodal Driver Visual Load Assessment and Prediction under Dynamic Lighting Environments

1. Problem Statement

Rapid transitions in roadway illumination, particularly in tunnels and shaded sections, significantly increase driver visual workload and compromise safety. These transitions induce physiological phenomena such as the "black hole" and "white hole" effects, leading to delayed visual adaptation, reduced contrast sensitivity, and impaired hazard recognition. Existing approaches to assessing driver visual load suffer from three primary limitations:

Reliance on Single Modalities or Static Thresholds: Many methods depend on isolated physiological indicators (e.g., pupil diameter) or manually predefined thresholds, failing to capture the complex, nonlinear interactions between driver behavior, environmental illumination, and vehicle dynamics.
Fragmented Analysis: Visual-load assessment and temporal prediction are often treated as independent tasks, preventing the full exploitation of the relationship between adaptive labeling and prediction.
Computational Constraints: While Transformer-based models show promise in capturing long-range temporal dependencies, their high computational complexity and memory requirements often hinder deployment in real-time Intelligent Transportation Systems (ITS).

There is a lack of a unified framework capable of simultaneously addressing adaptive visual-load classification, multimodal feature fusion, class imbalance, and computational efficiency under dynamic lighting conditions.

2. Methodology

The study proposes a unified multimodal framework comprising data acquisition, adaptive clustering for label generation, and a deep learning prediction architecture.

2.1 Data Acquisition and Preprocessing

Experimental Setup: Real-vehicle experiments were conducted at the Chang'an University proving ground involving 10 drivers (novice and experienced) across three environments: shaded sections, long straight roads, and tunnels.
Sensors: Data were synchronized from a Tobii Pro Glasses 3 eye tracker (50 Hz), a 360 Dash Cam (1 Hz), and a Tesan digital illuminometer.
Features: The dataset integrates eye-movement characteristics (gaze coordinates, pupil diameter, fixation duration, saccade amplitude/frequency), environmental illumination (Lux), vehicle speed, and driver demographics.
Preprocessing: The pipeline included noise removal (Kalman filtering), missing value interpolation (cubic spline), and Robust normalization to handle outliers. Temporal synchronization aligned all data streams to a unified timeline based on driving events. Subsequently, all modalities were resampled to a common frequency of 5 Hz: eye-tracking signals were downsampled, vehicle-speed measurements were linearly interpolated, and the original illumination sampling frequency was retained.

2.2 Visual Load Assessment (Labeling)

Instead of using predefined thresholds, the study employs an unsupervised learning approach to derive visual-load states:

Feature Extraction: Seven visual-load indicators (pupil area, change rates, MTPA, saccade amplitude, fixation duration, saccade frequency) were extracted.
Dimensionality Reduction: Principal Component Analysis (PCA) reduced these indicators to three principal components explaining 89.52% of the variance.
Clustering: A novel RaceGrid-OPTICS algorithm was developed. It combines:
- Progressive Grid Search: For coarse-to-fine parameter exploration.
- F-Race Optimization: To statistically evaluate and eliminate inferior parameter configurations.
- OPTICS: To identify clusters of varying densities without predefining the number of clusters.
Labeling: The clustering resulted in five interpretable visual-load levels (0–4), ranked by weighted scores derived from the principal components.

2.3 Visual Load Prediction Framework

The labeled data were used to train a temporal prediction model formulated as a supervised sequence-learning task.

Baselines: Four standard Deep Learning (DL) models were implemented: Fully Convolutional Network (FCN), Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and the standard Transformer.
Proposed Architecture (WA-PSRSMformer): Building on the Transformer, three enhancements were introduced:
1. Weighted Adaptive Loss (WAL): Addresses class imbalance by assigning higher penalties to underrepresented (high-load) classes.
2. Residual Sparse Attention (RSA): Replaces standard self-attention to reduce computational complexity ( $O(N^2)$ ) while preserving long-range dependencies and propagating attention information across layers.
3. Parameter Sharing (PS): Shares parameters across query, key, value projections, and feed-forward layers to reduce model redundancy and memory consumption.

3. Key Contributions

Multimodal Framework: Integration of synchronized eye-tracking, illumination, and vehicle dynamics data from real-world driving to assess visual load in dynamic lighting.
RaceGrid-OPTICS Clustering: A novel strategy combining PCA and adaptive parameter optimization (Grid Search + F-Race) to robustly identify visual-load states without manual thresholds, achieving high clustering quality.
WA-PSRSMformer Architecture: A Transformer-based model that simultaneously improves prediction accuracy, handles class imbalance, and reduces computational overhead through WAL, RSA, and PS mechanisms.
Empirical Validation: Comprehensive evaluation across three distinct roadway environments, demonstrating superior performance over state-of-the-art baselines.

4. Experimental Results

The framework was evaluated using accuracy, weighted precision/recall/F1-score, Mean Absolute Error (MAE), and rank correlation metrics (Kendall's Tau, Spearman's Rho).

Clustering Performance: RaceGrid-OPTICS outperformed default OPTICS, DBSCAN, and K-means, achieving a Silhouette Coefficient of 0.86, a Calinski–Harabasz Index of 159,532.83, and a Davies–Bouldin Index of 0.28.
Baseline Comparison: The standard Transformer consistently outperformed LSTM, CNN, and FCN across all environments. In tunnel scenarios, the Transformer achieved 85.54% accuracy compared to 80.72% (LSTM), 74.22% (FCN), and 72.42% (CNN).
Proposed Model Performance:
- WA-PSRSMformer achieved the highest overall performance, reaching 90.13% accuracy and an MAE of 0.1525 in tunnel-driving conditions.
- It demonstrated significant improvements over the baseline Transformer in tunnel environments (+5.37% accuracy) and straight roads (+1.71%).
- The model showed strong rank correlation with ground truth (Spearman's Rho = 0.8807 in tunnels).
Computational Efficiency: The proposed optimizations drastically reduced inference time and memory usage. In tunnel scenarios, WA-PSRSMformer reduced execution time by approximately 72.9% (from 43.46s to 11.77s) and memory usage by 16.3% compared to the baseline Transformer, making it suitable for real-time deployment.

5. Significance and Claims

The paper claims that the proposed framework provides an effective approach for real-time driver-state monitoring. By unifying adaptive clustering with an efficient deep learning architecture, the study addresses the critical gap in handling dynamic lighting environments where visual load fluctuates rapidly.

The authors emphasize that the WA-PSRSMformer successfully balances high prediction accuracy with low computational cost, a necessary condition for integration into Intelligent Transportation Systems (ITS) and Advanced Driver-Assistance Systems (ADAS). The study concludes that the framework offers a robust solution for predicting driver visual load under complex, changing illumination, thereby supporting the development of safer traffic safety systems. Future work is noted to focus on expanding scenario diversity and incorporating additional physiological signals (ECG, EEG).