SAM-Enhanced Segmentation on Road Datasets: Balancing Critical Classes in Autonomous Driving

Technical Summary: SAM-Enhanced Segmentation on Road Datasets

Problem Statement
Dense semantic segmentation is a fundamental requirement for autonomous driving perception, yet the development of robust models is hindered by a scarcity of multi-modal datasets with high-quality, pixel-level annotations. The Zenseact Open Dataset (ZOD), while rich in multi-sensor (camera-LiDAR) data from Northern European environments, provides only 2D bounding-box labels. This limitation restricts its utility for supervised semantic segmentation research. Furthermore, existing automated labeling approaches often fail in safety-critical domains due to hallucinations, boundary errors, or poor performance on small and occluded objects. Additionally, autonomous driving datasets suffer from extreme class imbalance, where safety-critical classes like pedestrians, cyclists, and traffic signs constitute less than 1% of total pixels, causing standard models to underperform on these rare categories.

Methodology
The authors propose a comprehensive pipeline to bridge the gap between sparse bounding boxes and dense segmentation masks, utilizing the Segment Anything Model (SAM) as a foundation.

1. SAM-Based Annotation Pipeline:

   • Preprocessing: Raw ZOD bounding boxes undergo filtering to remove noise (e.g., extremely small detections, elongated artifacts) and deduplication via IoU-based merging. A cap of 75 objects per frame is applied to balance completeness and efficiency.
   • Mask Generation: Filtered boxes serve as prompts for SAM's ViT-H model. To resolve overlapping masks, a priority-based fusion scheme is employed where vulnerable road users (pedestrians, cyclists) take precedence over signs and vehicles.
   • Output: The resulting masks are resized to 768×768 resolution. The pipeline generates two annotation variants: Camera-only (dense masks preserving full field of view) and LiDAR-only (masks projected strictly onto valid LiDAR return points to reflect sensor sparsity).
   • Curation: From over 100,000 processed frames, a manually curated subset of 2,300 frames (36% acceptance rate) was established to serve as a reliable benchmark.

2. Iseauto Dataset Enhancement:

   • For the Iseauto platform, a SAM-based edge refinement algorithm is applied to existing manual labels. This process uses SAM to perform conservative edge expansion and contraction, correcting boundary inaccuracies while preserving semantic intent.

3. Model Architectures:

   • The study evaluates two paradigms: CLFT (Camera-LiDAR Fusion Transformers), which uses cross-modal multi-head self-attention for joint reasoning, and DeepLabV3+, a CNN-based architecture with atrous spatial pyramid pooling extended for multi-modal fusion.
   • Model Specialization: To address class imbalance, specialized models are trained for single classes (Vehicle, Sign, Human) and pairwise combinations, alongside a parameter-merged ensemble approach.

4. Transfer Learning:

   • Bidirectional transfer learning experiments are conducted between the ZOD and Iseauto datasets to assess cross-domain generalization.

Key Contributions

• Annotation Pipeline: A novel SAM-based pipeline that converts sparse bounding boxes into dense, pixel-level semantic masks for the ZOD dataset.
• Benchmark Dataset: A curated pilot dataset of 2,300 manually verified frames, enabling the first segmentation benchmark on ZOD.
• Architectural Evaluation: A comparative analysis of Transformer-based (CLFT) and CNN-based (DeepLabV3+) architectures across diverse weather conditions (Day/Night, Fair/Rain/Snow).
• Class Specialization: Demonstration that specialized models and ensembles significantly improve segmentation performance for rare, safety-critical classes.
• Cross-Dataset Validation: Successful validation on the Iseauto platform and analysis of bidirectional transfer learning, showing accelerated convergence when initializing models from one dataset to another.

Results

• Performance on ZOD: The CLFT-Hybrid fusion model achieved the best baseline performance with 48.1% mIoU under Day Fair conditions. Transformer-based models demonstrated greater stability across adverse weather compared to DeepLabV3+.
• Rare Class Improvement: Specialized models yielded substantial gains for rare classes. For instance, a "Sign-only" model improved Sign IoU from 29.7% to 43.5%. An ensemble of specialized models reached 51.3% mIoU, though at a reduced inference speed (10.3 FPS vs. 27.8 FPS for the baseline).
• Annotation Impact: Evaluations revealed that using sparse LiDAR-projected annotations resulted in significantly lower IoU scores compared to dense SAM-generated masks, highlighting the value of dense supervision for inferring object extent.
• Iseauto Performance: On the Iseauto platform, the CLFT-Hybrid fusion model achieved 77.5% mIoU in Day Fair conditions, outperforming ZOD results due to clearer object boundaries and less complex scenes.
• Transfer Learning: Bidirectional transfer learning accelerated convergence, with models initialized from the opposite dataset reaching competitive performance within 20–30 epochs.

Significance and Claims
The paper claims to provide the first comprehensive analysis of dense multi-modal segmentation on the ZOD dataset, transforming it from a detection-only resource into a segmentation benchmark. The authors assert that SAM-derived representations transfer effectively across different sensor configurations and datasets, supporting transferable feature learning. While acknowledging limitations—such as SAM's struggles with small/distant objects and the high computational cost of transformer ensembles—the work establishes a foundation for reproducible research in dense perception. The authors position their ensemble approach primarily as an offline teacher for auto-labeling or knowledge distillation rather than a real-time solution, and they identify future work in scaling the curated dataset and improving small-object segmentation. All code, models, and processed datasets are released to support the research community.