Technical Summary: Cross-Level Sensor Fusion with Object Lists via Transformer for 3D Object Detection

Problem Statement

In autonomous driving, sensor fusion is critical for robust perception. Traditional fusion architectures are categorized into low-level (raw data), intermediate-level (features), and high-level (object lists/decisions). While low-level fusion theoretically offers optimal state estimation, practical constraints often prevent its use. Specifically, Vehicle-to-Everything (V2X) modules and smart sensors (e.g., MobileEye, Continental radars) frequently output only processed object lists rather than raw data due to bandwidth limitations and internal processing constraints.

Consequently, fusion is typically restricted to the high level, where distributed object detectors send state estimations to a central fuser. This approach suffers from data association errors and a loss of fine-grained information. Conversely, existing data-level fusion methods are designed for dense data (e.g., point clouds) and are not directly applicable to the sparse, structured nature of object lists. There is a lack of research exploring end-to-end learning methods that fuse highly abstract object list information with raw sensor data (specifically camera images) at the feature level.

Methodology

The authors propose an end-to-end cross-level fusion framework that integrates object lists with raw camera images using a Transformer-based architecture, specifically extending the PETRv2 3D object detection model. The core innovation lies in treating object lists not as final decisions but as priors to guide the feature extraction and detection process.

The methodology comprises three key components:

1. Object-Centric Query Denoising (QDN):

   • Inspired by DN-DETR, the system treats object lists as "noisy" ground truth containing estimation errors, false positives, and false negatives.
   • These object lists are converted into Denoising (DN) queries and concatenated with learnable detection queries.
   • Unlike standard DN-DETR which discards DN queries during inference, this framework allows detection queries to interact with denoising queries via self-attention. This enables the model to learn robust feature representations by reconstructing clean object states from noisy list inputs while enriching the queries with raw image features during cross-attention.
   • The attention mask between DN queries and learnable queries is removed to facilitate information flow, unlike the strict separation in DN-DETR.

2. Explicit Attention Directing via Deformable Gaussian Masks:

   • To explicitly guide the Transformer decoder, the authors introduce a deformable Gaussian mask derived from the positional and size priors of the object lists.
   • Object list coordinates are projected onto 2D image planes using camera parameters to generate a multi-target shared Gaussian mask ($M$).
   • This mask is integrated into the cross-attention mechanism of the Transformer decoder. The attention score is reweighted by adding the logarithm of the Gaussian mask ($\log G$) to the standard attention calculation:
     $$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + \frac{\log G}{\sqrt{d_k}}\right)V$$
   • This mechanism directs the model's attention toward specific target areas defined by the object list, accelerating convergence and improving localization.

3. Pseudo Object List Generation (POLG):

   • Since no public dataset contains object lists as a standalone modality, the authors developed a POLG module.
   • This module generates pseudo object lists from ground-truth bounding boxes by simulating realistic detector behaviors, including Gaussian noise in position/size, object drops (false negatives), false positives, and label misclassification.
   • Noise levels are configurable to simulate detectors with varying performance capabilities.

Key Contributions

The paper claims the following primary contributions:

1. First Cross-Level Fusion Framework: It is the first work to introduce an end-to-end Transformer-based framework for 3D object detection that fuses object lists with raw sensor data, moving beyond traditional high-level fusion.
2. Plug-and-Play Integration: The approach leverages Query Denoising and explicit attention directing (SMCA) to fuse modalities without requiring additional parameters or significant computational overhead, assuming a query-based architecture.
3. Pseudo Object List Generation (POLG): The development of a method to generate diverse, noisy object lists from ground truth, enabling the training and validation of cross-level fusion on standard datasets like nuScenes.

Experimental Results

The approach was evaluated on the nuScenes dataset using a PETRv2 baseline.

• Performance Improvement: The proposed model significantly outperformed the vision-only baseline. On the validation set, it achieved an NDS of 0.5685 and mAP of 0.5499, compared to the baseline's 0.4038 NDS and 0.4136 mAP.
• Ablation Studies:
  • Adding QDN alone improved NDS by 8.01% and mAP by 2.30% over the baseline.
  • Adding SMCA (explicit attention) on top of QDN further improved NDS by 5.94% and mAP by 4.30%.
  • The combination of both modules demonstrated synergistic benefits.
• Robustness to Modality Loss: When object lists were removed during inference (simulating a failure of the external sensor), the model's performance dropped significantly (mAP decreased by ~25%), indicating a strong reliance on the provided priors during the current training regime. However, the SMCA component provided slight mitigation against this degradation.
• Generalization: The model demonstrated robustness across different noise levels in synthetic lists and showed comparable performance when using object lists from real-world detectors (CenterPoint and PointPillars), validating the fidelity of the POLG noise parameters.

Significance and Claims

The paper positions this work as a novel step toward bridging the gap between high-level and low-level fusion. By treating object lists as informative priors rather than final decisions, the framework allows for the "early integration" of abstract information with raw sensor data.

The authors claim that this approach:

• Creates synergistic effects by combining the context of high-level information with the fine-grained details of raw data.
• Reduces state estimation errors and improves detection accuracy, particularly in scenarios where raw data from secondary sensors is unavailable.
• Offers a modular solution that can be integrated into existing DETR-like detectors.

The authors acknowledge limitations, noting that velocity estimation accuracy (mAVE) saw a slight trade-off, likely due to the current focus of denoising queries on position and size. They suggest future work should incorporate velocity-aware denoising and adaptive fusion strategies to handle missing object lists more robustly. The paper concludes that while the current method shows substantial promise, further comparison against traditional high-level fusion methods is necessary to fully assess practical applicability in autonomous driving.