💻 computer science

ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models

ViCo3D is a novel collaborative 3D object detection framework that bridges the modality gap between LiDAR and Vision Foundation Models by projecting point clouds into BEV images for DINOv2 feature extraction, thereby significantly enhancing feature-level collaboration and achieving state-of-the-art performance in V2X systems.

Original authors: Haojie Ren, Songrui Luo, Lingfeng Wang, Yan Xia, Yao Li, Jing Li, Lu Zhang, Jiajun Deng, Yanyong Zhang

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haojie Ren, Songrui Luo, Lingfeng Wang, Yan Xia, Yao Li, Jing Li, Lu Zhang, Jiajun Deng, Yanyong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a self-driving car, but instead of just relying on your own eyes (your LiDAR sensor), you have a team of friends helping you see the road. Some friends are in other cars, and some are standing on street corners with super-powered cameras. This is called V2X (Vehicle-to-Everything) collaboration. The goal is to share what everyone sees so you can spot a hidden pedestrian or a sneaky car better than you could alone.

But here's the problem: Your friends see the world differently. The car on the corner sees things from a high angle with a dense, clear view, while your car sees things from a low angle with a sparser, "grainier" view. Trying to mix these two different pictures is like trying to blend a high-definition photo with a sketchy doodle. Usually, the computer gets confused, and the teamwork doesn't work as well as it should.

The Big Idea: Borrowing a "Super-Brain" from 2D Images
The researchers behind this paper, ViCo3D, had a wild idea. They noticed that while LiDAR data (3D point clouds) is messy and hard to understand, Vision Foundation Models (VFMs)—specifically one called DINOv2—are already super-smart at understanding 2D images. These models were trained on millions of photos and know how to spot shapes, textures, and objects really well.

The team asked: What if we could trick DINOv2 into looking at our 3D LiDAR data as if it were a 2D picture?

How They Did It (The Magic Trick)

The Transformation: They took the 3D points from the LiDAR and flattened them onto a flat map (called a Bird's-Eye-View or BEV). They painted this map with three "colors" (channels): one for height, one for how shiny the object is, and one for how crowded the points are. Suddenly, the messy 3D dots looked like a regular image.
The Super-Brain: They fed this "LiDAR image" into DINOv2. The AI, which is an expert at 2D images, instantly started pulling out rich, smart features about the scene—things like "that looks like a car" or "that area is empty."
The Merging: But wait! DINOv2 is great at seeing, but it's not great at measuring exact distances (localization). So, the researchers didn't just replace their old system with DINOv2. Instead, they built a fusion engine. They took the "smart seeing" from DINOv2 and mixed it with the "precise measuring" from their original LiDAR system.
- First, they looked at the big picture (global context) to understand the whole scene.
- Then, they zoomed in to fix the tiny details (local refinement) so they didn't lose the exact shape of the objects.

What They Argued Against
The paper explicitly argues against a few common ideas:

Don't just force everyone to look the same: Some previous methods tried to force the car's view and the street-corner view to become identical. The authors say, "No!" Different views have different strengths. You want to keep those differences because they provide complementary information (one sees what the other misses).
Don't just swap the brain: You can't just throw away the LiDAR sensor and use DINOv2 alone. The paper shows that using only the vision model features leads to a massive drop in performance (it's like trying to drive using only a map without a speedometer). You need both.
Don't ignore the "modality gap": You can't just paste image-trained AI onto 3D data without a translator. The paper proves that without their special fusion steps, the image-trained model fails miserably on 3D tasks.

The Results: How Much Better Is It?
The team tested this on two real-world datasets: DAIR-V2X (real-world data) and V2XSet (simulated data).

The Win: ViCo3D beat every other method they tested. On the DAIR-V2X dataset, it achieved an AP@0.5 of 82.80 and an AP@0.7 of 71.39. (AP stands for Average Precision; higher is better).
The Collaboration Boost: This is the coolest part. When they added teamwork (collaboration), their method improved by 13.01 percentage points over a single car working alone.
Comparison: Other methods only improved by about 7 to 8 points. The authors note that their method delivers up to 1.8 times (or 1.89 times in the text) more benefit from teamwork than previous methods.

How Sure Are They?
The authors are very confident in these numbers because they ran extensive experiments on standard, public benchmarks. They didn't just guess; they measured it.

They proved that their method creates a "richer" feature space. Instead of relying on a few dominant channels (like a loud voice shouting over others), their method spreads the information out across many channels, allowing for more detailed and diverse understanding.
They showed that while their method makes the features less similar between the car and the street corner (lower cosine similarity), this is actually a good thing because it preserves the unique, helpful details each agent sees.

The Bottom Line
ViCo3D is a new way to make self-driving cars work better together. By turning 3D LiDAR data into a format that a 2D image expert (DINOv2) can understand, and then carefully mixing that "smart seeing" with precise 3D measurements, they created a system that spots objects more accurately and gains much more from teamwork than anyone else has before. It's not a magic fix for everything (they admit they still need to work on timing delays and adding cameras later), but for now, it's a huge step forward in making cars see the world as a team.

Technical Summary: ViCo3D

Problem Statement

LiDAR-based Vehicle-to-Everything (V2X) collaborative perception typically relies on fusing Bird's-Eye-View (BEV) features extracted from multiple agents (vehicles and infrastructure). However, current state-of-the-art approaches face two primary limitations:

Feature Heterogeneity: Existing LiDAR backbones are often trained from scratch on task-specific datasets, resulting in BEV representations that are heavily geometry-dominated and lack general semantic priors. This leads to significant discrepancies in feature distributions across agents due to differences in sensor configurations, observation viewpoints, and data densities (e.g., high-density infrastructure LiDAR vs. low-density vehicle LiDAR).
Modality Gap: While Vision Foundation Models (VFMs), such as DINOv2, have demonstrated superior capability in learning general-purpose visual representations from large-scale image data, directly adapting them to LiDAR-based 3D detection is non-trivial. The substantial gap between 2D image modalities and irregular 3D point clouds, combined with the lack of localization sensitivity in raw VFM features, leads to severe performance degradation if VFMs are used in isolation.

Existing methods attempting to align features across agents often prioritize feature consistency, potentially suppressing the complementary information inherent in diverse observations. The paper argues that effective collaboration should preserve these detection-relevant complementary cues rather than forcing strict feature alignment.

Methodology: ViCo3D

ViCo3D is a collaborative 3D object detection framework designed to bridge the image-point cloud modality gap by integrating Vision Foundation Models (specifically DINOv2) with LiDAR-based perception. The framework operates through three core stages:

1. Dual-Branch Feature Extraction

To leverage VFMs without discarding geometric precision, ViCo3D employs a dual-branch encoder for each agent:

Pillar Branch: Utilizes a standard PointPillars-based encoder to extract dense LiDAR BEV features ( $F_P$ ) directly from voxelized point clouds, preserving local geometric structures essential for 3D detection.
DINOv2 Branch: Projects raw LiDAR point clouds onto the BEV plane to generate three-channel "BEV images." These channels encode: (1) maximum normalized height, (2) normalized reflectance intensity, and (3) point density (log-scaled). These structured images are fed into a pretrained DINOv2 backbone to extract BEV-space visual features ( $F_D$ ) rich in semantic priors.

2. Feature Alignment and Multi-Scale Fusion

To integrate the geometric features ( $F_P$ ) and semantic features ( $F_D$ ) effectively, the framework introduces a multi-scale fusion module:

Feature Alignment: Learnable projection layers map features from both branches into a shared feature space to mitigate representation mismatches.
Low-Resolution Global Fusion (LRG): To avoid prohibitive memory costs associated with dense attention on high-resolution BEV maps, features are downsampled (4 $\times$ ). A PointPillars-dominant cross-attention mechanism is applied where the Pillar features act as queries and DINOv2 features as keys/values. This captures global contextual interactions efficiently.
High-Resolution Refinement Fusion (HRR): To recover fine-grained local details lost during downsampling, a high-resolution refinement stage operates on the original resolution. It explicitly models local relationships using difference ( $|F'_P - F'_D|$ ) and element-wise product ( $F'_P \odot F'_D$ ) terms to highlight discrepancies and complementary activation regions. A spatial dynamic gate adaptively balances the fused features with the original Pillar residual stream.

3. Ego-Centric Collaborative Fusion

Active Communication: Inspired by Where2comm, the framework employs an active communication mechanism where agents transmit only spatially critical regions (based on a confidence map) rather than full dense feature maps, reducing bandwidth.
Ego-Centric Aggregation: Upon receiving features from neighboring agents, the ego vehicle transforms them into its coordinate system. A gating mechanism adaptively adjusts the contribution of neighboring features relative to the ego feature, performing a residual update to incorporate complementary information while maintaining the ego-centric representation as the primary anchor.

Key Contributions

VFM-Enhanced LiDAR Perception: ViCo3D is the first work to explore adapting Vision Foundation Models (DINOv2) to enhance agent-wise LiDAR BEV representations for collaborative perception, successfully bridging the image-point cloud modality gap.
Multi-Scale Fusion Architecture: The paper proposes a novel fusion strategy combining Low-Resolution Global Fusion (for efficient context modeling) and High-Resolution Refinement (for preserving local details), enabling the effective integration of semantic priors with geometric features.
Complementarity over Consistency: Unlike prior methods that focus on feature distillation to enforce consistency, ViCo3D demonstrates that preserving diverse, complementary feature representations leads to superior collaborative gains. Empirical analysis shows ViCo3D achieves lower cross-view feature similarity than baselines while delivering higher detection performance.

Experimental Results

The framework was evaluated on two benchmarks: the real-world DAIR-V2X dataset and the simulated V2XSet dataset.

State-of-the-Art Performance: ViCo3D achieved the best performance among BEV-based detection methods.
- On DAIR-V2X, it reached 82.80 AP@0.5 and 71.39 AP@0.7.
- On V2XSet, it achieved 97.20 AP@0.5 and 93.23 AP@0.7.
Collaborative Gains: ViCo3D demonstrated the largest improvement over single-agent baselines. On DAIR-V2X at IoU=0.5, it improved the baseline by 13.01 percentage points, which is approximately 1.89 $\times$ the gain of INSTINCT and 1.60 $\times$ the gain of DI-V2X.
Ablation Studies:
- Using DINOv2 features alone resulted in poor performance (AP@0.5 $\approx$ 14.49), confirming the necessity of the dual-branch fusion.
- The combination of LRG and HRR modules was shown to be critical, with LRG improving recall and HRR recovering precision.
- Feature space analysis (PCA and channel energy) revealed that ViCo3D learns a more distributed feature subspace compared to baselines, supporting the hypothesis that it preserves complementary information rather than forcing alignment.

Significance and Claims

The paper claims that ViCo3D fundamentally shifts the paradigm of collaborative perception from seeking feature consistency to exploiting feature complementarity. By integrating the general semantic priors of Vision Foundation Models with the geometric precision of LiDAR backbones, the framework overcomes the heterogeneity issues inherent in V2X systems. The authors assert that this approach not only sets a new state-of-the-art for 3D detection but also provides a more robust mechanism for cross-agent information sharing, delivering significantly higher collaborative gains than existing intermediate-fusion methods. The work highlights that the "modality gap" between images and point clouds can be effectively bridged through structured BEV projection and multi-scale fusion, rather than requiring entirely new 3D foundation models.