💻 computer science

QueryOcc: Query-based Self-Supervision for 3D Semantic Occupancy

QueryOcc introduces a query-based self-supervised framework that learns continuous 3D semantic occupancy directly from 4D spatio-temporal queries and pseudo-point clouds or raw lidar data, achieving state-of-the-art performance on the Occ3D-nuScenes benchmark while maintaining real-time inference speeds.

Original authors: Adam Lilja, Ji Lan, Junsheng Fu, Lars Hammarstrand

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adam Lilja, Ji Lan, Junsheng Fu, Lars Hammarstrand

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car, but instead of seeing the world in 3D, your eyes only see flat, 2D pictures. To drive safely, your brain (or in this case, the car's computer) needs to build a mental 3D map of everything around it: where the road is, where the pedestrians are, and where the empty space is so you don't crash.

The paper introduces a new method called QueryOcc that teaches a computer to build this 3D map just by looking at a sequence of photos, without needing expensive human teachers to draw the map for it.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Teacher" is Too Expensive

Usually, to teach a computer to see in 3D, humans have to manually label millions of points in 3D space (like coloring in a 3D coloring book). This is incredibly slow and expensive.

Old Methods: Some previous AI tried to learn by "guessing" and checking if the 2D pictures looked right (like trying to solve a puzzle by only looking at the edge pieces). Others used laser scanners (LiDAR) but had to chop the world into tiny, rigid Lego blocks (voxels), which made the map look blocky and limited how far the car could "see."

2. The Solution: Asking Direct Questions (The "Query" Part)

Instead of trying to reconstruct the whole picture or chop the world into blocks, QueryOcc acts like a detective asking specific questions.

The Analogy: Imagine you are in a dark room and you want to know what's there. Instead of turning on the light and looking at the whole room, you stick a long, thin probe (a "query") into the air at a specific spot and ask, "Is there a car here?" or "Is this empty space?"
How it learns: The system asks thousands of these questions at different spots in 3D space and at different times. It checks its answers against "pseudo-labels" (smart guesses made by other AI models or data from laser scanners). If the system says "There's a car" but the data says "No," it learns from the mistake. This happens directly in 3D space, not by trying to recreate a 2D photo.

3. The Magic Trick: The "Folding Map" (Contractive Representation)

A car needs to see things right next to it (like a pedestrian stepping off the curb) in high detail, but it also needs to see far away (like a car 100 meters down the road).

The Problem: If you try to map everything with the same level of detail, your computer runs out of memory. It's like trying to draw a map of the whole world on a single piece of paper; the cities would be too small to see, or the paper would have to be the size of a football field.
The Solution: QueryOcc uses a "folding map" technique.
- Near the car: The map is unfolded and zoomed in. Every inch is detailed.
- Far away: The map is smoothly "compressed" or folded up. The distant mountains are squished together.
- Why it works: This allows the computer to remember the whole world (near and far) using the same amount of memory, while keeping the important details right where the car is driving.

4. The Result: Fast and Accurate

The paper claims that this method is a huge improvement over previous techniques:

Accuracy: It is 26% better at understanding the 3D shape and meaning of the scene compared to the best previous methods.
Speed: It runs fast enough to be used in real-time driving (about 11.6 frames per second), meaning it can keep up with a car moving on the highway.
Flexibility: It works even if the car only has cameras, or if it has cameras and laser scanners. It can mix and match these data sources.

Summary

QueryOcc is a new way for self-driving cars to learn how to see the 3D world. Instead of waiting for humans to draw the map or trying to build a giant, blocky Lego model, it asks direct questions about specific points in space and uses a "folding map" trick to see both close-up details and distant horizons without running out of memory. The result is a smarter, faster, and more accurate 3D vision system that learns on its own.

Technical Summary: QueryOcc

Problem Statement

Learning 3D scene geometry and semantics from images is a fundamental challenge for autonomous driving, yet large-scale 3D annotation is prohibitively expensive. Existing self-supervised approaches face significant limitations:

Rendering-based methods: These rely on 2D image reconstruction consistency. While scalable, they provide only indirect geometric signals where 3D structure emerges as a by-product of image synthesis rather than an explicit learning target. They often depend on externally estimated depth maps for stability.
Lidar-based methods: These provide explicit 3D supervision but rely on discretizing aggregated point clouds into predefined voxel grids. This approach limits spatial precision, scalability, and the ability to represent unbounded scenes efficiently.

The paper posits that direct supervision in continuous 4D space-time offers clearer geometric feedback for self-supervised 3D semantic occupancy learning than rendering-based or voxelized lidar methods.

Methodology: QueryOcc

QueryOcc is a query-based self-supervised framework that learns continuous 3D semantic occupancy directly from multi-view images (and optionally lidar) without rendering losses or voxel aggregation.

1. Model Architecture

The framework follows a modular four-stage pipeline:

Image Encoding: Multi-view images are processed by a pre-trained backbone to extract dense features. Camera calibration parameters are encoded via a lightweight MLP to provide camera-specific context.
Lift-Contract-Splat (LCS): This is the core innovation for handling unbounded scenes. It extends the Lift-Splat-Shoot (LSS) formulation with three key components:
- Point-wise Encoding: Instead of splatting raw features, the model explicitly encodes visibility, uncertainty, and temporal cues using a learnable function. This allows the representation of both occupied and unobserved regions.
- Log-Linear Depth Binning: To balance near-field resolution with far-field coverage, depth bins use exponentially increasing spacing outside a near-field region ( $d_{near}$ ), rather than uniform spacing.
- Axis-Aligned Contraction: To maintain constant memory and computation regardless of scene range, an axis-aligned contraction function compresses distant regions while preserving high-resolution detail near the ego vehicle. This avoids the quadratic memory growth associated with expanding BEV grids.
BEV Feature Processing: The contracted BEV features are processed using ResBlocks and multi-scale deformable attention to capture local and global dependencies, with dynamic convolutions to adapt to the local scaling of the contraction.
Unified Query Decoder: A lightweight decoder predicts occupancy and semantics for arbitrary 4D queries ( $q = [x, y, z, t]^\top$ ). Queries are contracted to align with the BEV space, encoded by an MLP, and fused with interpolated BEV features. The decoder predicts offsets to extract features from nearby locations (mimicking deformable attention) before outputting the final predictions.

2. Self-Supervised Training

The framework operates directly in continuous 4D space-time, generating supervision from point clouds observed across time.

Supervision Sources:
- Camera-only: Pseudo point clouds are generated by lifting pixels into 3D using metric depth predicted by a vision foundation model (VFM). These are paired with semantic pseudo-labels or dense VFM features.
- Camera-Lidar: Real lidar point clouds serve as explicit geometric supervision, projected into camera views to retrieve semantic labels or features.
- Unified: A combination of both sources.
Query Generation: Supervision is applied via spatio-temporal 4D queries sampled from the point clouds:
- Negative (Unoccupied) Queries: Sampled along the ray from the sensor origin up to the observed point.
- Positive (Occupied) Queries: Sampled within a buffer zone behind the observed point.
Loss Function: The model is trained using a combination of binary cross-entropy for occupancy, categorical cross-entropy for semantics, and L1 loss for VFM feature distillation.

Key Contributions

QueryOcc Framework: A flexible, query-based framework that learns continuous 3D semantic occupancy from multi-view images without rendering losses or lidar voxelization.
Unbounded Contractive Representation: A novel scene representation that preserves near-field detail while smoothly compressing distant regions, enabling long-range supervision under constant memory. This is coupled with a "lift-contract-splat" mechanism to lift image features into this space.
State-of-the-Art Performance: The method achieves superior results on the self-supervised Occ3D-nuScenes benchmark, outperforming prior methods by 26% in semantic RayIoU while maintaining real-time inference (11.6 FPS).

Experimental Results

Evaluated on the Occ3D-nuScenes benchmark:

Performance: QueryOcc achieves a semantic RayIoU of 23.6% (and 25.8% for the enhanced QueryOcc+ variant), surpassing the previous best self-supervised method (GaussianFlowOcc) by 26%. It also improves occupancy RayIoU by 25%.
Efficiency: The model runs at 11.6 FPS on an A100 GPU with a total latency of 86ms, significantly faster than rendering-based NeRF approaches (e.g., 1.2 FPS for SelfOcc).
Robustness: The model performs well across various classes, particularly excelling in small and thin categories (e.g., traffic cones, pedestrians). It is robust to the choice of image encoder (e.g., ConvNeXt-Base or ResNet-50).
Ablation Studies:
- Direct query-based supervision significantly outperforms rendering-based supervision (23.6 vs. 15.0 RayIoU), indicating that 2D reconstruction provides weaker geometric constraints.
- All proposed components (log-linear bins, contraction, point encoding) contribute to performance gains.
- Combining camera and lidar supervision yields the best results, effectively fusing dense semantic cues with accurate geometry.
- Performance scales with higher input resolution and additional datasets (e.g., Argoverse 2), demonstrating strong data efficiency and cross-domain generalization.

Significance

The paper claims that QueryOcc establishes a new benchmark for self-supervised 3D scene understanding from cameras. Its significance lies in demonstrating that direct 4D query-based self-supervision is a robust foundation for large-scale learning, offering clearer geometric feedback than rendering-based methods. By eliminating the need for voxelization and rendering losses, the framework achieves a balance of high geometric accuracy, computational efficiency, and the ability to reason over unbounded scenes. The work highlights that continuous 4D supervision enables strong learning of both geometry and semantics as explicit targets, rather than as by-products of image synthesis.