VALISENS: A Validated Innovative Multi-Sensor System for Cooperative Automated Driving

Technical Summary: VALISENS – A Validated Innovative Multi-Sensor System for Cooperative Automated Driving

Problem Statement
Reliable perception remains a critical bottleneck for Connected Automated Vehicles (CAVs) operating in complex real-world environments. While existing multi-sensor solutions enhance local robustness, they are fundamentally constrained by limited sensing ranges, line-of-sight occlusions caused by other objects, and the potential for individual sensor failures. Furthermore, adverse weather and varying lighting conditions can significantly degrade the performance of optical sensors. The paper addresses the need to extend perception capabilities beyond the individual vehicle through Vehicle-to-Everything (V2X) collaboration between CAVs and intelligent infrastructure, aiming to create a dynamic, decentralized sensor network that mitigates occlusions and improves situational awareness, particularly for Vulnerable Road Users (VRUs).

Methodology and System Architecture
The authors propose VALISENS, a validated cooperative perception system that integrates onboard and roadside sensors within a unified multi-agent framework. The system architecture is modular, separating perception by sensor modality before fusing data at two distinct levels:

1. Hardware Deployment:

   • Infrastructure: Two testbeds were utilized. The primary site in Dresden features six cameras (including thermal and RGB), a 64-layer LiDAR, and a 4x4 MIMO radar. A secondary site in Ingolstadt includes 22 sensor masts. A mobile "Rover" platform was also employed, equipped with RGB, thermal, and LiDAR sensors.
   • Vehicles: Two test vehicles were used. Vehicle One utilized a dual-camera setup (Seek infrared and Basler RGB), two roof-mounted Velodyne VLP-16 LiDARs, and six IBEO Scala LiDARs. Vehicle Two featured an InnovizOne LiDAR and a DTC xNAV650 inertial navigation system. Both vehicles utilized Cohda MK6 On-Board Units (OBUs) for V2X communication.
   • Sensors: The system fuses LiDAR, radar, RGB cameras, and thermal cameras. Thermal cameras are specifically highlighted for enhancing VRU detection under challenging lighting.

2. Perception and Fusion Strategy:

   • Modality Processing: LiDAR data is processed using an AI-based algorithm for 3D detection and tracking. Radar data is fused into a Bird's-Eye View (BEV) and post-processed to reconstruct 3D coordinates. RGB and thermal data are processed independently using the YOLO framework and fused via a late fusion strategy using Non-Maximum Suppression (NMS) and ByteTrack for object tracking.
   • Intra-Entity Fusion: Within a single platform (vehicle or infrastructure), a late fusion approach aggregates object-level information. A track-to-track fusion algorithm aligns data spatially and temporally (using constant velocity models), associates detections across modalities using Euclidean distance and Intersection-over-Union (IoU), and fuses them using an adapted Kalman filter.
   • Inter-Entity Fusion: To enable cooperation, the system utilizes the Collective Perception Message (CPM) standard defined by ETSI. Perception results are broadcast via V2X. Receiving entities transform the data into their local coordinate frames, align them, and fuse them with local observations using the same track-to-track pipeline.

3. Sensor Condition Monitoring:

   • A dedicated module assesses sensor health to detect anomalies before system degradation. The system focuses on detecting permanent and non-permanent faults, including adverse weather, lens blockage (dirt, ice, fog), and internal imager defects.
   • A classifier trained on raw camera images (combining reliability-test recordings and public datasets) predicts fault classes. The ConvNeXt architecture was selected as the optimal model, achieving high accuracy in anomaly detection.

Key Contributions
The paper outlines three primary contributions:

1. Real-World Deployment: The design, implementation, and deployment of a cooperative perception system in a real-world testbed involving two CAVs and two infrastructure units.
2. Comprehensive Evaluation: Quantitative and qualitative evaluation using both public datasets (R-LiViT, V2X-Seq) and real-world experiments, demonstrating the specific benefits of incorporating thermal cameras and roadside sensors.
3. Validation of Monitoring and Communication: Validation of the integrated sensor monitoring module and V2X communication stack in a real-world setting, confirming the feasibility of early anomaly detection and reliable data exchange.

Experimental Results

• Perception Performance:
  • Public Datasets: On the R-LiViT dataset, RGB-Thermal fusion improved nighttime pedestrian Average Precision (AP50) from 49.85% to 53.26%. On the V2X-Seq-SPD dataset, inter-entity fusion increased pedestrian AP50 by approximately 30% compared to vehicle-only sensing (reaching 30.86).
  • Real-World Experiments: In the dedicated testbed, inter-entity fusion achieved a pedestrian AP50 of 65.75 with an average recall of 97.60%. The system improved pedestrian situational awareness by up to 18% compared to vehicle-only sensing.
• Sensor Monitoring: The sensor monitoring module achieved over 97% accuracy in detecting anomalies, with the ConvNeXt model demonstrating an F1-macro score of 0.981.
• Communication: V2X communication tests showed a maximum range exceeding 1 km under line-of-sight conditions. The one-way delay was measured at a maximum of 5.0 ms. Data rates for CPMs were efficient, with a maximum of 33 kB/s even at 10 Hz update frequencies.

Significance and Claims
The paper claims that VALISENS demonstrates the effectiveness of extending multi-sensor fusion beyond a single vehicle through V2X-enabled collaboration. By integrating thermal cameras and roadside sensors, the system significantly reduces occlusions and expands the effective perception range, directly contributing to improved safety for VRUs. The successful validation of the sensor monitoring module highlights a pathway toward more resilient Cooperative Intelligent Transport Systems (C-ITS) capable of self-diagnosis and maintaining reliability in the face of sensor degradation or environmental challenges. The authors position this work as a step toward supporting future C-ITS applications by establishing a dynamic, decentralized sensor network that offers a comprehensive and continuously updated understanding of the surrounding environment.