🤖 AI

HilDA: Hierarchical Distillation with Diffusion for Advancing Self-Supervised LiDAR Pre-trainin

HilDA is a self-supervised pre-training framework for LiDAR backbones that leverages hierarchical distillation from Vision Foundation Models and a temporal occupancy diffusion objective to achieve state-of-the-art performance in 3D object detection, scene flow, and semantic occupancy prediction by better capturing semantic and geometric information.

Original authors: Maciej Wozniak, Jesper Ericsson, Hariprasath Govindarajan, Truls Nyberg, Thomas Gustafsson, Patric Jensfelt, Olov Andersson

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maciej Wozniak, Jesper Ericsson, Hariprasath Govindarajan, Truls Nyberg, Thomas Gustafsson, Patric Jensfelt, Olov Andersson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot driver how to see the world. You have two types of "eyes":

The Camera: It sees the world like a human does—full of colors, textures, and clear labels (like "that's a stop sign" or "that's a dog"). It's great at understanding what things are, but it's bad at knowing exactly where they are in 3D space or how they are moving.
The LiDAR: This is a laser scanner that creates a precise 3D map of the world. It knows exactly where every point is, but it sees the world as a cloud of dots without any labels. It doesn't know if a dot is a tree or a car; it just knows it's there.

The problem is that teaching the LiDAR to recognize objects usually requires humans to manually label millions of 3D dots, which is slow, expensive, and impossible to cover every possible scenario (like a dog running out from behind a truck in the rain).

Enter HilDA: A new method that teaches the LiDAR to "learn from the Camera" without needing human labels. The authors call it HilDA (Hierarchical Distillation with Diffusion). Here is how it works, using simple analogies:

1. The "Master Chef" and the "Apprentice" (Hierarchical Distillation)

Usually, when teaching a student (the LiDAR model) from a master (the Camera model), you only look at the final result. It's like a cooking teacher only showing the student the finished cake and saying, "Make this."

The authors realized this is inefficient. A master chef doesn't just make the cake; they mix the batter, check the temperature, and frost the layers in a specific order.

The Old Way: The LiDAR only tried to copy the final "cake" (the final layer of the camera's brain).
The HilDA Way: The LiDAR watches the entire process. It learns from the "mixing stage," the "baking stage," and the "frosting stage" (multiple layers of the camera's brain).
The Global Context: It also teaches the LiDAR to understand the "vibe" of the whole scene. Just as a chef knows the difference between a wedding cake and a birthday cake based on the whole table, HilDA teaches the LiDAR to recognize if it's on a highway or in a residential neighborhood, not just looking at individual dots.

2. The "Time-Traveling Crystal Ball" (Temporal Occupancy Diffusion)

Knowing what things are is great, but a self-driving car also needs to know what will happen next. The camera is good at recognizing a car now, but it doesn't inherently understand how that car will move in the next second.

To fix this, HilDA adds a "crystal ball" training exercise:

The Game: The LiDAR is shown the road at time $T$ and time $T-1$ . It is then asked to "predict" what the road will look like at time $T+1$ .
The Diffusion Trick: Instead of just guessing, the model plays a game of "denoising." Imagine the future road is covered in static noise. The model has to slowly wipe away the noise to reveal the clear future road.
Why it helps: This forces the LiDAR to learn the rules of physics and motion. It learns that cars move smoothly, pedestrians walk, and buildings stay still. It learns the "geometry" of the world, not just the labels.

3. The Result: A Super-Perceptive Driver

By combining these two ideas—learning the step-by-step process of recognizing objects (from the camera) and learning to predict the future (from the diffusion game)—HilDA creates a LiDAR model that is incredibly smart.

What the paper claims this achieves:

Better Accuracy: The LiDAR makes fewer mistakes in identifying objects, even in tricky situations like a person standing on top of a truck or a scooter driver in the rain.
Less Data Needed: It works very well even when the team only has a tiny amount of labeled data (as little as 1% of what other methods need).
Robustness: It handles bad weather (snow, fog) and sensor errors much better than previous methods.
Versatility: This single trained model works great for many different driving tasks, not just one. It helps with:
- 3D Object Detection: Finding cars and people.
- Scene Flow: Understanding how fast and in what direction things are moving.
- Semantic Occupancy: Knowing exactly which parts of the 3D space are empty, solid, or occupied by specific objects.

In short, HilDA is like giving the robot driver a mentor who shows them the whole cooking process and a crystal ball to predict the future, resulting in a driver that understands both what things are and where they are going.

Technical Summary: HilDA – Hierarchical Distillation with Diffusion for LiDAR Pre-training

1. Problem Statement

Autonomous driving (AD) systems require robust spatial perception in dynamic environments. While LiDAR provides high-fidelity geometric data, training 3D models typically necessitates massive, labeled datasets. Manual annotation is not only costly but also fails to cover the combinatorial diversity of real-world geometry and motion. To address this, the community has turned to Knowledge Distillation (KD) from Vision Foundation Models (VFMs) as a self-supervised pre-training strategy for LiDAR backbones.

However, existing cross-modal distillation approaches suffer from two primary limitations:

Final-Layer Restriction: Current methods treat VFMs as black boxes, distilling only the features from the final layer. This ignores the progressive semantic refinement and intermediate representations distributed across the VFM's layers, which are crucial for capturing "how" features are constructed. Furthermore, these methods often discard the global context encoded in the VFM's CLS token.
Lack of Spatiotemporal Supervision: VFMs trained on images excel at semantic recognition but lack explicit 3D geometric and temporal cues. Pure semantic distillation leaves the 3D structure underconstrained. While some prior works introduce occupancy prediction, they typically rely on discriminative objectives that model local marginals rather than joint scene distributions, failing to enforce global spatiotemporal coherence.

2. Methodology: HilDA Framework

The authors propose HilDA (Hierarchical Distillation with Diffusion), a self-supervised pre-training framework that combines hierarchical distillation with a temporal occupancy diffusion objective. The framework processes temporal sequences of three frames ( $t_{-1}, t_0, t_1$ ), using the past and present for feature extraction and the future frame as a target for occupancy prediction.

2.1 Hierarchical Distillation

HilDA moves beyond single-layer distillation by capturing the semantic evolution of the VFM through two mechanisms:

Multi-Layer Distillation: Instead of aligning only the final output, HilDA aligns intermediate layers of the student LiDAR network with corresponding layers of the frozen VFM teacher. This is achieved via a dense-to-sparse strategy where 3D points are projected onto 2D camera images using calibration parameters. A lightweight, layer-specific MLP aligns feature dimensions, and cosine distance is minimized over valid point-pixel correspondences across multiple layers. This allows the student to learn the progressive construction of visual concepts.
Global Context Distillation: To capture scene-level semantics, the framework aligns the VFM's CLS token (which encodes global scene information) with a learnable 3D global context token in the student network. The student's global token is generated by applying a dedicated MLP to the final layer features followed by global max-pooling. The alignment is enforced via Mean Squared Error (MSE), acting as a scene-level regularizer.

2.2 Temporal Occupancy Diffusion

To address the lack of explicit 3D geometry and temporal dynamics, HilDA introduces an auxiliary self-supervised task: Temporal Occupancy Diffusion.

Formulation: The task is cast as conditional Denoising Diffusion Probabilistic Model (DDPM) generation. Given LiDAR sweeps at $t_{-1}$ and $t_0$ , the student network encodes them into a shared dense Bird's Eye View (BEV) history feature map.
Objective: The model learns to denoise a corrupted future occupancy target ( $x_{occ}$ at $t_1$ ) conditioned on the history. The training objective combines a noise prediction loss (standard diffusion) with a reconstruction loss for the occupancy map.
Benefit: This generative approach encourages the model to internalize spatial manifolds and surface hierarchies through iterative refinement, promoting spatiotemporal consistency and object permanence without requiring manual labels.

2.3 Total Objective

The framework is trained end-to-end using a weighted sum of three losses:
$\mathcal{L}_{total} = \omega_{ds}\mathcal{L}_{distill} + \omega_{gl}\mathcal{L}_{cls} + \omega_{df}\mathcal{L}_{diffusion}$
where $\mathcal{L}_{distill}$ is the multi-layer point-pixel loss, $\mathcal{L}_{cls}$ is the global context loss, and $\mathcal{L}_{diffusion}$ is the temporal occupancy diffusion loss.

3. Key Contributions

Hierarchical Distillation Strategy: A novel approach that distills both multi-layer intermediate features and global CLS tokens from VFMs, enabling the LiDAR student to learn progressive semantic refinement and holistic scene context.
Temporal Occupancy Diffusion: An auxiliary generative objective that casts future occupancy prediction as conditional diffusion, effectively injecting multi-scale geometric and motion cues into the LiDAR encoder.
State-of-the-Art Performance: The integration of these components yields representations that outperform prior distillation methods across multiple benchmarks.

4. Experimental Results

The authors evaluated HilDA on the nuScenes dataset for pre-training, transferring the learned backbones to various downstream tasks without re-pre-training.

3D Semantic Segmentation: HilDA achieves State-of-the-Art (SOTA) results on nuScenes, SemanticKITTI, and Waymo Open Dataset. It shows significant gains in data-scarce scenarios (1%–10% labels), outperforming methods like CleverDistiller, LiMoE, and ScaLR. It also demonstrates superior domain generalization on unstructured off-road (Rellis-3D) and scribble-labeled (ScribbleKITTI) datasets.
Robustness: On the corrupted nuScenes-C benchmark, HilDA exhibits superior resilience to weather (snow, rain) and sensor failures (beam echo, cross-sensor), achieving the lowest mean Corruption Error (mCE).
3D Object Detection: When used as an initialization for PointRCNN, HilDA improves mAP on KITTI and nuScenes, particularly in complex scenarios involving long-range and heavy occlusion.
Semantic Occupancy & Scene Flow: HilDA outperforms baselines in 3D and 4D semantic occupancy prediction and scene flow estimation (Argoverse 2). The diffusion auxiliary task specifically improves performance on dynamic objects and motion estimation.

5. Significance and Claims

The paper claims that HilDA successfully bridges the gap between semantic richness and geometric/temporal robustness in LiDAR pre-training. By moving beyond the "black-box" final-layer distillation and incorporating a generative diffusion prior, the framework creates a representation that captures both the "semantic what" (from VFMs) and the "geometric where" (from diffusion).

The authors emphasize that this approach reduces dependence on large-scale annotated data while producing features that are transferable across diverse domains and tasks. They note a limitation: point-pixel distillation remains sensitive to LiDAR-camera misalignment, though HilDA partially mitigates this via alignment-free global context distillation and temporal diffusion. Future work is suggested to explore learned matching strategies for layer-to-layer distillation and extending diffusion beyond BEV representations.