💻 computer science

Easy3D-Labels: Supervising Semantic Occupancy Estimation with 3D Pseudo-Labels for Automotive Perception

This paper introduces Easy3D-Labels, a method that generates 3D pseudo-ground-truth labels using Grounded-SAM and Metric3Dv2 to enable direct 3D supervision for semantic occupancy estimation, thereby eliminating the high computational costs of novel view synthesis while significantly improving localization accuracy and performance on the Occ3D-nuScenes dataset.

Original authors: Seamie Hayes, Ganesh Sistu, Tim Brophy, Ciaran Eising

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seamie Hayes, Ganesh Sistu, Tim Brophy, Ciaran Eising

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car, but instead of just looking at the road in front of you, your car needs to build a complete, 3D holographic map of the entire world around it. It needs to know exactly where the sidewalk is, where a pedestrian might step out, and how high a building's overhang is. This is the job of Semantic Occupancy Estimation.

However, teaching a computer to build this 3D map is incredibly hard. Here is the problem the paper solves, explained simply:

The Problem: The "Flat Map" vs. The "3D World"

Traditionally, self-driving cars looked at the world like a flat map (Bird's Eye View). This is like looking at a shadow of a 3D object. You can see the shape, but you lose the height and depth.

The Issue: If you only have a flat shadow, you might think a tall tree is a low bush, or you might miss a pedestrian hiding behind a car because the "shadow" looks empty.
The Solution: We need a full 3D representation, like a digital Lego city made of tiny blocks (voxels), where every block knows if it's a road, a car, or a person.

The Old Way: The Expensive "Render and Guess" Game

To teach a computer to build this 3D Lego city without hiring humans to label every single block (which is too expensive and slow), researchers used a tricky method:

They would guess what the 3D world looks like.
They would use a computer to "render" (draw) what that 3D world would look like from the camera's angle.
They would compare that drawing to the actual photo to see if they were right.

The Flaw: This is like trying to learn how to sculpt a statue by constantly squinting at a 2D photograph of your work, trying to guess the 3D shape. It's computationally heavy (slow), confusing, and often leads to mistakes like "ghost objects" (seeing a car twice) or getting the depth wrong.

The New Solution: "Easy3D-Labels"

The authors of this paper, Easy3D-Labels, decided to stop playing the "render and guess" game. Instead, they built a 3D cheat sheet (pseudo-labels) that tells the computer exactly what the 3D world looks like, right from the start.

Here is how they made this cheat sheet, using a creative analogy:

1. The "Magic Camera" (Grounded-SAM & Metric3Dv2)

Imagine you have two super-powered assistants looking at the car's camera photos:

Assistant A (The Artist): Looks at the photo and draws a perfect outline of everything it sees (a car, a tree, a person). This is Grounded-SAM.
Assistant B (The Ruler): Looks at the same photo and measures exactly how far away every single pixel is. This is Metric3Dv2.

2. The "3D Projector"

Instead of just keeping these drawings and measurements flat on the photo, the system takes every single point the Artist drew and the Ruler measured, and projects them into 3D space.

Analogy: Imagine taking a 2D drawing of a tree and using a laser projector to turn it into a real, floating 3D tree in the air.
The result is a cloud of 3D points that already knows what it is (a tree) and where it is (10 meters away).

3. The "Time-Lapse" (Temporal Aggregation)

One photo isn't enough. The car is moving, so it takes many photos over a few seconds.

The system takes all these 3D point clouds from the last few seconds and stacks them together.
Analogy: It's like taking a time-lapse video of a busy street. By combining all the frames, the system fills in the gaps. If a pedestrian was hidden behind a bus in one frame, they appear in the next. The system merges them to create a complete, dense 3D map without "ghosts" or duplicates.

The Result: "EasyOcc"

The authors built a new model called EasyOcc that uses this 3D cheat sheet to learn.

No more rendering: It doesn't need to waste time drawing 2D pictures to check its work. It just compares its 3D Lego city directly to the 3D cheat sheet.
Better Safety: Because it understands depth and 3D space better, it is much better at spotting vulnerable road users like pedestrians and cyclists.
Huge Gains: When they added this cheat sheet to existing models, their performance jumped by 45%. When they used it to train their own model (EasyOcc), it beat many complex, expensive models.

Why This Matters

Think of it like learning to play a video game.

The Old Way: You have to play the game, die, look at a replay, guess what went wrong, and try again. It takes forever.
The Easy3D Way: You are given a "Walkthrough Guide" (the 3D labels) that shows you exactly where the enemies are and where the obstacles are before you even start playing. You learn faster, make fewer mistakes, and get better at the game much quicker.

In short: This paper gives self-driving cars a "3D cheat sheet" generated by AI, allowing them to understand the world in 3D much faster, cheaper, and more accurately than before, making our roads safer for everyone.

1. Problem Statement

Automated vehicle perception relies heavily on accurate 3D scene understanding for safety-critical tasks like motion planning and forecasting.

Limitations of Existing Methods: Traditional representations like Bird's Eye View (BEV) collapse vertical information, leading to ambiguities in 3D localization.
Challenges in Self-Supervised Learning: While self-supervised semantic occupancy estimation reduces the need for expensive manual LiDAR annotations, current approaches rely on 2D pseudo-labels. To train on these 2D labels, models must employ Novel View Synthesis (NVS) (e.g., NeRF or 3D Gaussian Splatting) to render 3D volumes back into 2D image space.
Drawbacks of Current Approaches:
- High Computational Cost: NVS and rendering during training are computationally intensive and memory-heavy.
- Geometric Biases: Relying on 2D rendering often leads to biases toward nearby objects, inaccurate depth estimation, and object duplication.
- Performance Gap: There remains a significant performance gap between supervised (LiDAR-based) and self-supervised models.

2. Methodology

The authors propose Easy3D-Labels, a method to generate 3D pseudo-ground-truth labels directly, bypassing the need for 2D rendering during training.

A. Easy3D-Labels Generation Pipeline

The labels are generated offline using foundation models and temporal aggregation:

Semantic Point Cloud Generation:
- Input: Multi-view camera images from the dataset.
- Semantics: Grounded-SAM is used to generate pixel-level semantic segmentation maps.
- Depth: Metric3Dv2 (Giant variant) is used to generate metric depth maps.
- Projection: 2D pixels are projected into 3D global coordinates using camera intrinsics and extrinsics (Equation 1), creating a semantic point cloud.
Densification (Temporal Aggregation):
- To increase point density and fill occluded regions, semantic point clouds from previous time steps ( $T-\Delta$ ) are aggregated.
- Dynamic Object Filtering: Dynamic objects (vehicles, pedestrians) are removed from historical frames to prevent "ghosting" or duplication, while static objects (roads, sidewalks) are retained.
- The aggregated points are transformed into the ego-vehicle coordinate frame.
Voxelization:
- The densified point cloud is voxelized into a 3D grid (0.4m resolution, bounded by $[-40m, 40m, 5.4m]$ ).
- Occupancy Thresholding: A voxel is marked as occupied only if it contains a minimum number of points (e.g., 10) to filter noise. The semantic label is assigned based on the majority class.

B. Model Integration & Loss Function

The generated 3D labels serve as direct supervision in 3D space, eliminating the need for rendering losses.

Pseudo-Loss ( $L_{Pseudo}$ ): A combination of Cross-Entropy ( $L_{CE}$ ) and Geometry Loss ( $L_{Geometry}$ ), which includes geometric scale, semantic scale, and Lovász-Softmax losses.
Integration: This loss is added as an auxiliary term to existing self-supervised models (SelfOcc, OccNeRF, GaussianOcc).
EasyOcc: The authors introduce a streamlined model that solely relies on Easy3D-Labels for supervision. It removes complex components like pose estimation, novel view synthesis, and multi-frame photometric consistency, using only a ResNet-101 encoder, bilinear sampling, and 3D CNNs.

3. Key Contributions

Easy3D-Labels: A novel pipeline generating 3D pseudo-labels using Grounded-SAM and Metric3Dv2, enabling direct 3D supervision without NVS.
Significant Performance Gains:
- Applied to OccNeRF on the Occ3D-nuScenes dataset, Easy3D-Labels increased mIoU by 45% and RayIoU by 49%.
- Pedestrian Segmentation: In SelfOcc, pedestrian IoU improved by over 600%, highlighting the method's ability to detect vulnerable road users.
EasyOcc Model: A simplified architecture achieving 15.7 mIoU on Occ3D-nuScenes using only 3D pseudo-labels, outperforming complex Gaussian-based methods without requiring LiDAR or inference-time foundation models.
Reduced Complexity: The approach eliminates the computational overhead of rendering and depth estimation during training, significantly reducing training time (e.g., EasyOcc trains in 1h 25m vs. 5h 8m for OccNeRF).

4. Results and Evaluation

The models were evaluated on Occ3D-nuScenes and OpenOccv2 using mIoU (mean Intersection over Union) and RayIoU (a ray-based metric that penalizes over-prediction and depth errors).

Quantitative Improvements:
- mIoU: EasyOcc (15.7) surpassed Gaussian-based baselines like GaussTR (13.8) and TT-Occ (16.7, but with heavy inference costs).
- RayIoU: OccNeRF with Easy3D-Labels achieved a RayIoU of 15.5 (vs. 10.4 baseline), a 49% improvement, indicating better depth accuracy and reduced object duplication.
- Dynamic Classes: Significant gains were observed for dynamic objects (cars, pedestrians, bicycles), which are critical for safety.
Qualitative Analysis:
- Object Duplication: Baseline models often duplicated objects or predicted occupancy in empty space; Easy3D-Labels significantly reduced this.
- Occluded Regions: Temporal aggregation allowed the model to correctly estimate regions beneath the ego-vehicle and building overhangs that were missing in ground truth or other models.
- Smoothness: The voxel-based EasyOcc produced smoother, more continuous scene representations compared to the fragmented outputs of Gaussian-based methods.
Ablation Studies:
- Temporal Aggregation: Using 13 temporal samples provided the best balance, showing diminishing returns beyond this point.
- Loss Components: The geometry loss and cross-entropy were both critical; removing them degraded performance.
- Empty Class Handling: Excluding the "empty" class from the Lovász-Softmax loss calculation improved mIoU by 0.5 points.

5. Significance and Conclusion

Paradigm Shift: The paper demonstrates that 3D supervision is superior to 2D rendering-based supervision for semantic occupancy, offering a more direct and computationally efficient learning signal.
Scalability: By removing the dependency on LiDAR ground truth and complex rendering pipelines, this approach makes high-quality 3D perception more accessible and scalable for autonomous driving.
Safety Impact: The dramatic improvement in detecting vulnerable road users (pedestrians, cyclists) directly addresses critical safety gaps in current perception systems.
Future Work: The authors suggest integrating LiDAR into the label generation pipeline for further refinement and testing robustness in adverse weather conditions (rain, fog).

In summary, Easy3D-Labels bridges the gap between self-supervised and supervised semantic occupancy estimation by leveraging foundation models and temporal data to create high-quality 3D training targets, resulting in state-of-the-art performance with reduced computational cost.