Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

Technical Summary: Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

Problem Statement
Autonomous driving perception systems face significant challenges in understanding dynamically changing environments characterized by varying weather conditions, poor illumination, and unpredictable traffic. Traditional approaches often rely on single-task models or single-sensor modalities (e.g., RGB cameras), which can fail under specific conditions like low light or heavy rain. Furthermore, existing multi-task learning (MTL) approaches often suffer from imbalanced learning when handling heterogeneous tasks with different loss functions, leading to suboptimal performance or training instability. Additionally, combining multiple recent models to cover all perception tasks results in high computational costs, large model sizes, and excessive GPU memory usage, hindering real-time deployment.

Methodology
The authors propose a compact deep multi-task learning (MTL) model designed to execute multiple perception tasks in a single forward pass without relying on external models.

• Architecture and Sensor Fusion: The model utilizes an encoder-decoder architecture with specific encoders for different input modalities: four RGB cameras, four Dynamic Vision Sensors (DVS), and one 360° LiDAR. The system processes four views (Front, Left, Right, Rear) for RGB and DVS, and a top view for LiDAR.

  • Intermediate Fusion: Inspired by intermediate fusion strategies, the model employs two bottleneck layers (convolution blocks) to fuse extracted features. The first bottleneck aggregates latent space from RGB, DVS, and LiDAR encoders to support Semantic Segmentation (SS), Depth Estimation (DE), and LiDAR Segmentation (LS). The second bottleneck aggregates information from these tasks to perform Bird's Eye View Projection (BEVP).
  • Task-Specific Decoders: The network branches into task-specific decoders for SS, DE, LS, and BEVP. Skip connections are used to enhance performance by combining features from the encoder and decoder.
  • LiDAR Pre-processing: To leverage vertical information, the authors propose a 15-layer LiDAR representation. This stacks 14 channels based on point cloud height (z-coordinate) and one channel containing flattened point clouds, preserving vertical structure better than single-layer projections.

• Adaptive Loss Weighting (MGN): To address the imbalanced learning issue inherent in MTL, the authors introduce a Modified Gradient Normalization (MGN) algorithm.

  • Unlike standard GradNorm which updates weights frequently, MGN updates loss weights only once per epoch (at the final step) to reduce computational overhead.
  • The algorithm dynamically adjusts loss weights based on the relative inverse training rate of each task, ensuring that tasks with different characteristics (e.g., SS vs. BEVP) are learned at similar rates.
  • The total loss is a weighted sum of Huber loss (for DE) and a combination of Binary Cross-Entropy and Dice loss (for SS, LS, and BEVP).

• Datasets: Experiments were conducted on three CARLA simulation datasets (varying in map diversity and data volume) and one real-world dataset (nuScenes-lidarseg). For nuScenes, the model was adapted to use only RGB inputs (no DVS) and synthetic ground truths were generated from LiDAR point clouds for SS and DE tasks.

Key Contributions

1. Compact Multi-Task Model: A unified model that performs multi-view semantic segmentation, depth estimation, LiDAR segmentation, and bird's eye view projection simultaneously, eliminating the need for multiple separate models.
2. Adaptive Loss Weighting Strategy: A modified GradNorm algorithm tailored for a multi-modal, multi-view architecture with two shared bottlenecks, effectively balancing the learning process across heterogeneous tasks.
3. Enhanced LiDAR Representation: A 15-layer pre-processing technique for LiDAR point clouds that retains vertical height information, demonstrating improved performance in segmentation and depth estimation tasks.
4. Comprehensive Evaluation: A rigorous ablation study and comparative analysis against a combination of state-of-the-art single-task and multi-task models (PolarNet, Chen et al., and GradNorm) across simulation and real-world scenarios.

Results

• Performance: The model variant trained with 15-layer LiDAR data and the MGN algorithm (15L+MGN) achieved the best overall performance, indicated by the lowest Total Metric (TM) score and Metric Variance (MV) across most datasets.
  • On simulation datasets, the 15L+MGN model outperformed the combination of recent models (Chen et al. + PolarNet + GradNorm) in terms of TM scores on datasets A, B, and nuScenes, though the combination performed slightly better on dataset C (attributed to PolarNet's superior LiDAR segmentation capabilities).
  • The 15-layer LiDAR input consistently improved LiDAR segmentation (LS) performance and generally enhanced SS and DE tasks compared to the 1-layer input.
  • The MGN algorithm successfully balanced the training dynamics, reducing the discrepancy between tasks (lower MV), even if it occasionally resulted in a trade-off where specific tasks (like DE or SS) performed slightly better with static weights.
• Efficiency: The proposed model is significantly more efficient than the combination of recent models. It possesses fewer than 2% of the total parameters of the combined baseline models.
  • Inference Speed: The model achieves approximately 54 FPS on simulation datasets and 65 FPS on the real-world nuScenes dataset.
  • Resource Usage: It requires significantly less GPU memory (approx. 20 MB vs. over 1000 MB for the combination) and has a much smaller model size.

Significance
The paper claims that its primary significance lies in demonstrating that a compact, single-model approach can achieve superior or comparable performance to a combination of specialized models while drastically reducing computational resources. By integrating multi-sensor fusion and an adaptive loss weighting strategy, the model provides a robust solution for autonomous driving perception that is faster, more memory-efficient, and better suited for real-time deployment on edge devices. The work highlights that balancing multi-task learning through adaptive weighting and leveraging rich sensor representations (like vertical LiDAR data) are critical for achieving a holistic and efficient understanding of the driving environment. The authors emphasize that their model maintains consistent performance across diverse simulation and real-world conditions, validating its potential for practical application.