💻 computer science

OccAnyScene: Towards Unified Indoor-Outdoor 3D Occupancy Predictio

The paper introduces OccAnyScene, a novel pixel-frustum-centered Gaussian framework that achieves state-of-the-art unified 3D semantic occupancy prediction across diverse indoor and outdoor scenes by leveraging a pretrained depth model to adaptively handle varying camera configurations, spatial scales, and semantic taxonomies.

Original authors: Junjie Liu, Wanshui Gan, Zitong Dai, Guiping Cao, Yan Li, Ke Chen, Dongmei Jiang, Xiangyuan Lan, Jianguo Zhang

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junjie Liu, Wanshui Gan, Zitong Dai, Guiping Cao, Yan Li, Ke Chen, Dongmei Jiang, Xiangyuan Lan, Jianguo Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D map of the world, but you only have a flat, 2D camera to look at it. This is the daily challenge for robots and self-driving cars trying to understand their surroundings. They need to know not just where the walls and trees are, but also what's hiding behind them. Scientists call this "3D occupancy prediction." Think of it like a digital sculptor that takes a flat photo and tries to guess the shape of the entire room or street, filling in the invisible parts with invisible blocks. For a long time, these digital sculptors were like specialists who could only work in one specific type of room. A sculptor trained to map a messy bedroom couldn't suddenly switch to mapping a busy highway, because the rules for measuring distance, the size of the blocks, and even the names of the objects were completely different.

This paper tackles the big question: Can we build one single "universal sculptor" that can handle both a cozy indoor living room and a sprawling outdoor city street without getting confused? The authors argue that the old way of doing things—having separate models for every different environment—is too clunky and hard to manage. They want a system that is flexible enough to switch between a small room with tiny, detailed blocks and a giant street with huge, coarse blocks, all while using the same brain.

The team behind this research introduces a new approach called OccAnyScene. Instead of trying to force a rigid grid onto every scene, they treat every single pixel (the tiny dots that make up a photo) as if it were a flashlight beam shining into the world. They realized that a pixel doesn't just point to a single line in space; it actually covers a cone-shaped area, like a flashlight beam that gets wider the further it travels. They call this a "frustum."

Here is how their magic trick works. First, they use a pre-trained "depth expert" (a model that is already really good at guessing how far away things are) to get a rough idea of the scene. Then, they use a clever two-step process. The first step, which they call Pixel-Aligned Frustum Feature Aggregation, gathers clues from the surrounding area to figure out what might be hiding behind the visible objects. It's like looking at the shadow of a chair to guess where the wall behind it is. The second step, Frustum-Parameterized Gaussian Construction, turns these clues into 3D shapes. Instead of guessing the exact size and position of a 3D object in absolute terms (which would be a nightmare to get right for both a tiny toy and a giant truck), they let the size of the 3D shape grow or shrink based on how wide the "flashlight beam" is at that specific distance. This makes the system naturally adapt to whether it's looking at a small room or a huge highway.

The results are impressive. When they tested this single model on a dataset of indoor rooms (Occ-ScanNet), it achieved a score of 59.92% accuracy in identifying objects and structures. When they switched it to a dataset of outdoor driving scenes (SurroundOcc-nuScenes), it scored 23.06%. Crucially, the paper shows that this single, flexible model performs just as well as the specialized models that were trained only on one type of scene. This suggests that we don't need a different robot brain for every environment; one adaptable brain can learn to see the world in 3D without losing its touch. The authors found that by letting the 3D shapes "breathe" and change size based on the camera's view, they could fill in the gaps of what's hidden behind objects, creating a complete picture of the world whether it's a bedroom or a city street.

Technical Summary: OccAnyScene

Problem Statement

3D semantic occupancy prediction is essential for holistic scene understanding in autonomous driving and embodied AI. However, existing methods are predominantly scene-specific, designed for fixed protocols regarding spatial ranges, voxel resolutions, camera configurations, and semantic taxonomies. This specialization hinders the development of general-purpose 3D perception.

The paper introduces Cross-Scene 3D Semantic Occupancy Prediction, a new task setting requiring a single model to handle heterogeneous indoor and outdoor scenes simultaneously. This setting presents a fundamental challenge: achieving metric-consistent yet scene-adaptive image-to-3D lifting.

Metric Inconsistency: Indoor scenes (e.g., rooms) require fine-grained, short-range perception, while outdoor scenes (e.g., streets) require coarse-grained, long-range perception.
Parameterization Difficulty: Existing Gaussian-based methods struggle because the absolute offsets and scales required for occlusion completion vary substantially across domains. A shared absolute parameterization is difficult to learn due to differing camera intrinsics and scene depths.

Methodology: OccAnyScene

To address these challenges, the authors propose OccAnyScene, a pixel-frustum-centered Gaussian framework built upon a pretrained depth foundation model (e.g., DepthAnythingV2/V3). The framework treats the finite 3D frustum associated with each feature pixel as the basic unit for constructing a continuous scene representation, rather than relying on fixed voxel grids or single rays.

The architecture consists of two primary modules:

1. Pixel-Aligned Frustum Feature Aggregation (PFFA)

This module constructs a camera-aware frustum query for each feature pixel to reason about occluded regions.

Input: It takes dense geometry features ( $F_{geo}$ ) from the depth foundation model's DPT head and visual tokens from its ViT encoder.
Initialization: A pixel query is initialized using geometric cues and the camera ray direction derived from intrinsics.
Context Aggregation: The module restores sequential ViT tokens to a 2D grid and performs pixel-aligned deformable cross-attention. This aggregates surrounding semantic and structural cues around the feature pixel, producing a frustum query that encodes visible-surface geometry, viewing geometry, and local context.

2. Frustum-Parameterized Gaussian Construction (FPGC)

This module decodes each frustum query into multiple Gaussian primitives ( $K$ Gaussians per pixel) with positions and scales constrained by the predicted pixel depth and frustum geometry.

Depth Anchoring: To ensure metric consistency across datasets with varying intrinsics, the model predicts a canonical-camera surface depth ( $D_{surf}$ ) using a lightweight MLP. This serves as a common depth anchor.
Positioning:
- Depth Increments ( $\Delta d_{p,k}$ ): Instead of placing all Gaussians on the surface, the model predicts depth increments to extend the representation from visible surfaces into occluded regions.
- Subpixel Offsets ( $\Delta u_{p,k}$ ): To account for the coarse nature of downsampled feature pixels, bounded subpixel offsets allow Gaussian centers to shift laterally within the corresponding pixel frustum.
Scale Parameterization: To avoid learning unstable absolute scales, the model uses a frustum-relative scale. It calculates a scalar frustum scale reference ( $b_{p,k}$ ) based on the metric cross-section of the frustum at the Gaussian's depth. The decoder then predicts a dimensionless relative scale ( $\hat{s}_{p,k}$ ), which is multiplied by the reference to determine the final Gaussian size. This adapts spatial supports to different camera parameters and scene ranges automatically.
Output: The resulting Gaussians are mapped to scene-specific semantic taxonomies via learnable taxonomy matrices and splatted onto the target voxel grid.

Key Contributions

Task Definition: The paper formally introduces Cross-Scene 3D Semantic Occupancy Prediction, a setting where a single model must jointly learn from heterogeneous indoor and outdoor scenes with different protocols.
Framework Design: The proposal of OccAnyScene, a pixel-frustum-centered Gaussian framework. It unifies occupancy prediction through:
- PFFA: For constructing frustum queries that reason about occlusions.
- FPGC: For scene-adaptive Gaussian construction that parameterizes positions and scales relative to the frustum geometry, enabling stability across varying scales and intrinsics.
Performance: Extensive experiments demonstrate that OccAnyScene achieves state-of-the-art (SOTA) results on both indoor (Occ-ScanNet) and outdoor (SurroundOcc-nuScenes) benchmarks. Crucially, the cross-scene model performs comparably to separately trained scene-specific models, proving that a unified model can handle diverse environments without significant performance degradation.

Experimental Results

Indoor (Occ-ScanNet): The model achieves 59.51% mIoU in the cross-scene setting using the DAv3 encoder, nearly matching the scene-specific performance (59.92% mIoU) and outperforming other cross-scene adaptations (e.g., SplatSSC† drops significantly).
Outdoor (SurroundOcc-nuScenes): The model achieves 22.87% mIoU in the cross-scene setting using the DAv3 encoder, again comparable to its scene-specific counterpart (23.06% mIoU) and superior to adapted baselines.
Efficiency: OccAnyScene-DAv2 demonstrates high efficiency with 98.2M parameters, 86.4ms inference time, and 670 MiB memory usage on an RTX 4090, reducing memory usage by over 80% compared to prior works like EmbodiedOcc and SplatSSC.
Ablation Studies:
- Removing PFFA or FPGC leads to significant performance drops, confirming the necessity of frustum-based feature aggregation and parameterization.
- The "frustum-relative scale" and "depth increments" are critical for handling metric variations and occlusion completion.
- Increasing the number of Gaussians per pixel ( $K$ ) from 1 to 3 yields marginal gains, suggesting that occlusion completion is largely achieved through the collective contribution of neighboring pixels rather than deep stacking within a single frustum.

Significance and Claims

The paper claims that OccAnyScene successfully addresses the difficulty of cross-scene occupancy prediction by moving away from fixed spatial representations toward a geometric reference that adapts to camera geometry and scene range.

The authors assert that their approach allows for the consolidation of room-scale and street-scale occupancy prediction into a single feed-forward model. This eliminates the need for maintaining and switching between separate models for different environments, thereby simplifying deployment and scaling for autonomous systems operating in diverse settings. The work demonstrates that with the correct geometric parameterization (pixel-frustum-centered), a unified model can achieve performance comparable to specialized models while maintaining high computational efficiency.

Limitations Acknowledged:
The authors note that current evaluation is limited to two datasets (Occ-ScanNet and SurroundOcc-nuScenes) and does not yet demonstrate generalization to arbitrary unseen scenes. Additionally, the pixel-frustum representation inherently covers only regions within the camera fields of view; gaps between cameras in surround-view setups are filled by a small set of supplementary spatial queries, which have a minimal impact on aggregate metrics but indicate a limitation for covering regions entirely outside the camera view.