💻 computer science

CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids

CLFTv2 is an efficient camera-LiDAR fusion framework for semantic segmentation that replaces global ViT attention with a hierarchical Swin-based encoder and lightweight decoder to significantly improve vulnerable road user detection and throughput while reducing computational costs compared to query-based and global-attention alternatives.

Original authors: Toomas Tahves, Mauro Bellone, Raivo Sell

Published 2026-09-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Toomas Tahves, Mauro Bellone, Raivo Sell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Autonomous vehicles rely on a constant stream of sensory data to navigate the world, but seeing is not the same as understanding. To drive safely, a car must distinguish between a painted line on the road and a pedestrian stepping off the curb, even when that person is small, distant, or partially hidden. Cameras provide rich color and texture, allowing the vehicle to recognize shapes and signs, but they cannot measure distance with certainty. Lidar, a laser-based scanning system, provides precise three-dimensional geometry, mapping the shape of objects in space, but it often produces sparse data that becomes increasingly empty as distance grows. For years, engineers have tried to fuse these two distinct streams of information into a single, reliable picture. The challenge lies in processing this massive amount of data quickly enough for real-time driving while ensuring that the most critical targets, such as people and cyclists, are never missed.

Researchers Toomas Tahves, Mauro Bellone, and Raivo Sell have introduced a new approach to this problem called CLFTv2. Their work focuses on a specific type of artificial intelligence architecture designed to combine camera and Lidar data more efficiently than previous methods. In the past, some leading systems used a mechanism known as a global attention network, which attempts to look at every part of an image and every part of the 3D scan simultaneously to find connections. While powerful, this method is computationally heavy, requiring immense processing power that can slow down a vehicle's computer. The authors proposed replacing this global view with a hierarchical, window-based system. Instead of scanning the entire scene at once, their model breaks the image down into smaller, shifting windows, processing local details first and then building up a broader understanding. This structure mimics how human vision often works, focusing on immediate surroundings before integrating them into a larger context.

The team tested this new framework across three major driving datasets representing different environments: the Zenseact Open Dataset from Sweden, the Waymo Open Dataset from the United States, and the ISEAuto dataset from Estonia. These datasets vary in weather, road conditions, and how the data was labeled, providing a rigorous test for the system's adaptability. The results showed that CLFTv2 successfully identified vulnerable road users with high reliability. On the Waymo dataset, the system achieved a performance score of 61.7 percent, a significant improvement over earlier versions of similar technology. On the ZOD dataset, it reached 53.5 percent, a notable jump that specifically improved the detection of pedestrians and traffic signs. Perhaps most importantly, the new system did this while using far less computing power than its competitors. It required roughly half the energy of some existing high-performance models and processed data more than twice as fast, making it a more practical choice for the limited hardware found inside a real car.

However, the study also revealed a nuanced trade-off in how these systems process information. While the window-based approach proved highly efficient and effective on most datasets, the researchers found that on the Waymo dataset, which contains extremely dense and detailed Lidar scans, a system with a global, all-seeing view still held a slight advantage in fusing the two data types. The local windows of the new system sometimes struggled to fully connect the dots in such dense geometric environments compared to the global method. This suggests that while the new architecture is a major step forward for efficiency, the perfect solution might eventually require a hybrid approach that combines the speed of local processing with the broad reach of global attention.

The researchers also examined how the system handles different types of data supervision. In some datasets, the ground truth labels were generated by other algorithms rather than human annotators, introducing a layer of uncertainty. Despite this, the new model learned to generalize well, showing that its ability to combine visual and geometric cues is robust even when the training data is imperfect. The study confirms that for the specific task of identifying small, critical objects like pedestrians, a carefully designed, lightweight fusion system can outperform much heavier, more complex models. By prioritizing the detection of vulnerable road users, the system ensures that safety remains the primary goal, even as the computational demands of autonomous driving continue to grow. The work demonstrates that in the race to build safer self-driving cars, sometimes a focused, efficient approach is more effective than a brute-force attempt to see everything at once.

Technical Summary of CLFTv2: Efficient Camera-LiDAR Fusion for Semantic Segmentation via Hierarchical Feature Pyramids

Problem Statement
Semantic segmentation for autonomous driving faces a critical challenge: the reliable detection of Vulnerable Road Users (VRUs), such as pedestrians, whose pixel footprint in perspective images is often minuscule (0.3–0.4% of total pixels) yet vital for safety. While cameras offer dense color and texture, they lack metric depth; conversely, LiDAR provides 3D geometry but yields increasingly sparse point clouds at distance. Existing multi-modal fusion approaches often struggle with computational efficiency or fail to extract reliable features for distant, small objects. Specifically, Vision Transformer (ViT) based encoders, while effective for cross-modal context, suffer from quadratic computational complexity and typically maintain a single fixed spatial resolution, making the generation of multi-scale feature pyramids required for small-object detection computationally expensive.

Methodology
The authors propose CLFTv2, a hierarchical camera-LiDAR fusion framework that operates entirely within the 2D perspective-projection domain. The architecture replaces the global attention mechanisms of standard ViTs with a Swin Transformer backbone and a lightweight, feature pyramid network (FPN)-style decoder.

Hierarchical Backbone: CLFTv2 employs two weight-shared Swin Transformer encoders (Siamese architecture) to process aligned RGB camera images and projected LiDAR coordinate maps. The LiDAR input is encoded as a dense 3-channel tensor (X, Y, Z coordinates) to ensure structural compatibility with standard vision backbones. The Swin architecture utilizes shifted-window self-attention, restricting computation to local $M \times M$ windows. This reduces complexity from quadratic to linear relative to image resolution. Patch merging layers generate a four-stage feature pyramid, naturally providing multi-scale representations essential for detecting small objects.
Coarse-to-Fine Residual Fusion: The decoder integrates features from both modalities across four spatial scales (strides 4, 8, 16, 32).
- Channel Projection: Features from each scale are projected to a unified dimension ( $D=256$ ) via $1 \times 1$ convolutions.
- Modality Refinement: Modality-specific Residual Convolution (ResConv) units refine the camera and LiDAR features independently.
- Fusion Mechanism: The framework employs a coarse-to-fine accumulation strategy. Starting from the coarsest scale (Stage 4), refined features are summed element-wise. At each subsequent finer scale, the accumulated representation from the previous coarser stage is bilinearly upsampled and added to the current scale's refined features before passing through a shared output ResConv unit. This design propagates global semantic context while preserving fine-grained spatial details.
Segmentation Head: The final fused representation is processed by a lightweight head involving a $3 \times 3$ convolution, Batch Normalization, ReLU, and a $1 \times 1$ convolution, followed by upsampling to the original input resolution.

Key Contributions

Architecture: Introduction of CLFTv2, a hierarchical Swin-based fusion architecture with a lightweight per-scale residual decoder. It avoids the heavy computational overhead of universal query-based decoders (e.g., Mask2Former) by utilizing native multi-scale geometric cues.
Efficiency vs. Accuracy: Demonstrated that CLFTv2 achieves competitive accuracy against state-of-the-art query-based decoders while requiring significantly fewer GFLOPs (1.4 $\times$ fewer than a Swin-based Mask2Former adaptation) and delivering 2.2 $\times$ higher throughput.
Modality-Isolation Insights: A study revealing a practical trade-off: while Swin's hierarchical structure improves small-object resolution, its local-window attention struggles to fully exploit dense, localized LiDAR returns compared to the global receptive field of ViT. This suggests that global attention remains superior for dense point-cloud alignment (as seen in the Waymo dataset), whereas local attention offers efficiency for sparse scenarios.
Generalization: Evidence that learned multi-modal representations generalize across differing sensor characteristics and geographical datasets, with cross-dataset initialization accelerating convergence.

Experimental Results
The framework was evaluated on three autonomous driving datasets: ZOD (Sweden, pseudo-labeled), Waymo (USA, dense LiDAR), and ISEAuto (Estonia, high-quality labels).

ZOD Performance: CLFTv2-Large achieved 53.5% mIoU, improving pedestrian IoU from 35.5% (previous CLFT model) to 44.9%. It outperformed the prior CLFT model and matched the accuracy of Mask2Former-Large (52.5% mIoU) with significantly lower computational cost.
Waymo Performance: On the Waymo dataset, characterized by dense LiDAR returns, the ViT-based CLFT family outperformed CLFTv2 (68.3% vs. 61.7% mIoU). This confirms that global attention is superior for fusing dense geometric data, while local attention (Swin) offers a more efficient alternative for sparse conditions.
ISEAuto Performance: Query-based models (MaskFormer/Mask2Former) led on this dataset, though CLFTv2 remained competitive (73.3% mIoU).
Efficiency: At the Tiny scale, CLFTv2-Tiny ran at 22.0 ms (81.8 ms for Large) with 187 MB memory usage, compared to 48.4 ms and 314 MB for Mask2Former-Tiny.
Safety Metrics: CLFTv2 demonstrated high recall for VRUs (e.g., 96.1% pedestrian recall on Waymo Day Fair), prioritizing safety-critical detection over precision in some configurations.

Significance and Claims
The paper claims that hierarchical local-attention fusion offers a scalable, efficient alternative to global-attention and query-based decoders for real-time on-vehicle perception. The authors argue that for closed-set semantic segmentation, the complex training overhead of query-matching mechanisms (Hungarian matching, deep supervision) is not always necessary to achieve state-of-the-art results, particularly under sparse or pseudo-labeled supervision.

The work highlights a nuanced design choice: while global attention (ViT) is superior for dense geometric alignment, the proposed Swin-based approach provides a favorable accuracy-efficiency trade-off, making it suitable for resource-constrained, real-time intelligent transportation systems. The authors conclude that architectural complexity is not universally beneficial and that lightweight residual fusion can perform on par with computationally demanding decoders when tailored to specific sensor densities and supervision qualities.

More like this