💻 computer science

RayOcc: Occlusion-Aware Ray Occupancy Estimation via Gaussian Mixture Intensity

RayOcc is a camera-only 3D semantic occupancy prediction framework that addresses depth ambiguity and occlusion by modeling rays as multi-label existence problems using Gaussian mixture intensities and Poisson event formulations, thereby enabling the coexistence of multiple occupied hypotheses and achieving state-of-the-art performance on the nuScenes benchmark.

Original authors: Junho Kim, Seongwon Lee

Published 2026-07-21

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Junho Kim, Seongwon Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a busy city street, but you only have a set of flat, 2D photos taken from different angles. This is the daily challenge for self-driving cars: they need to understand the full 3D world—where the cars are, where the sidewalks end, and how far away the trees are—using only cameras, not expensive laser scanners. This field is called "3D semantic occupancy prediction." Think of it like trying to guess the shape of a hidden object inside a foggy box just by looking at its shadow. The tricky part is "occlusion," which is just a fancy word for things blocking other things. If a big truck is in front of a small fence, a camera sees the truck, but the fence is hidden behind it. In the real world, both the truck and the fence exist along the same line of sight, but for a long time, computer programs struggled to see both at once. They were forced to pick just one "winner" for every line of sight, often forgetting the hidden fence or guessing the wrong distance.

This is where a new method called RayOcc steps in to change the game. Instead of forcing the computer to choose a single "best guess" for what lies along a line of sight, RayOcc allows the system to imagine multiple possibilities at the same time. It treats the space along a camera's view as a series of potential "events" that could happen, rather than a single choice. By using a clever mathematical trick involving "Gaussian mixtures" (which you can think of as overlapping clouds of probability) and a "Poisson event" model (like counting how many raindrops might hit a specific spot), RayOcc can say, "There's a high chance of a truck here, and also a high chance of a fence behind it." The researchers tested this on the nuScenes dataset, a massive collection of driving scenes, and found that their method creates much cleaner, more accurate 3D maps than previous approaches, especially in crowded or tricky scenes where things are hidden behind other things.

The Problem: The "One Winner" Rule

To understand why RayOcc is special, we first need to look at how computers usually try to "see" in 3D. Imagine you are looking down a long, straight hallway. In the old way of doing things, the computer would look at that hallway and say, "Okay, I see a wall at 10 meters." It would pick that single distance and ignore everything else. If there was actually a chair at 5 meters and a painting at 15 meters, the computer might get confused, or it might just pick the chair and pretend the painting doesn't exist.

This happens because most existing methods use a rule called "softmax normalization." Think of this like a pie. If you have a pie, and you cut it into slices for every possible distance, the total size of the pie must always equal 100%. If you give a huge slice to the "chair at 5 meters," there is almost no pie left for the "painting at 15 meters." The computer is forced to make the slices compete against each other. If the chair is big, the painting gets starved. This works okay if there is only one object, but in a real city, a single line of sight often passes through a car, then a fence, then a building. Forcing these objects to fight for a single slice of the pie leads to broken, messy 3D models where thin objects disappear or geometry looks fragmented.

The Solution: Letting Multiple Hypotheses Coexist

RayOcc throws out the "one pie" rule. Instead of asking, "Which object is here?" it asks, "How many objects might be here, and how strong is the evidence for each?"

The authors introduce a concept called Gaussian Mixture Intensity. Imagine you are shining a flashlight down that hallway again. Instead of painting a single spot on the wall, the flashlight sprays a mist of "intensity" along the entire beam. This mist can have multiple peaks. One peak might be thick and bright where the truck is, and another peak can be slightly dimmer but still clearly visible where the fence is behind it. Because this "intensity" isn't forced to add up to a fixed total (like the pie), the truck doesn't steal the light from the fence. They can both exist strongly at the same time.

To turn this mist into a decision, RayOcc uses a Poisson event formulation. This is a statistical way of thinking about random events. Imagine you are waiting for raindrops to hit a specific square on the sidewalk. If the "intensity" of rain is high, the chance of at least one drop hitting that square is very high. RayOcc treats every small slice of the hallway as a square on the sidewalk. If the "intensity" of an object passing through that slice is high enough, the system decides, "Yes, something is occupying this space." This allows the system to mark multiple slices along the same line as "occupied" without them fighting each other.

From Clouds to 3D Blobs

Once the system has figured out where the "intensity" peaks are, it needs to build the actual 3D model. The paper uses a technique involving 3D Gaussian primitives. You can think of these as fuzzy, glowing 3D blobs that float in space. Each blob has a center, a size, a color (or semantic label like "car" or "road"), and a transparency.

In previous methods, the computer would pick just one blob for every line of sight, usually based on the single "best" depth guess. RayOcc, however, looks at its multiple intensity peaks. If it sees a strong peak for a truck and a weaker but clear peak for a fence, it doesn't have to choose. It can spawn a blob for the truck and a blob for the fence. This is called adaptive component-wise sampling. It's like a chef who, instead of picking just one ingredient for a soup, adds a few different ones based on how strong their flavors are, creating a richer, more complex dish.

These blobs are then refined and "rasterized" (painted onto a grid) to create the final 3D map that the self-driving car uses to navigate.

The Results: Cleaner Maps, Better Driving

The researchers tested RayOcc on the nuScenes benchmark, a standard test set containing 1,000 driving scenes from Boston and Singapore. They compared their method against other top-tier camera-only systems.

The results showed that RayOcc achieved the state-of-the-art performance among the Gaussian-based methods they tested. Specifically:

It reached an overall IoU (Intersection-over-Union) of 34.84.
It achieved a mIoU (mean IoU) of 22.03.

To put this in perspective, the previous best method using a similar approach (VG3T) scored 34.06 IoU and 21.74 mIoU. RayOcc improved these numbers, meaning its 3D maps matched the real world more closely.

The paper highlights that this improvement is most noticeable in occluded scenes—places where objects are hidden behind others. In visual comparisons, RayOcc was able to reconstruct the full shape of a truck even when it was next to a thin fence, whereas other methods tended to "spill" the truck's shape into the fence or lose the fence entirely. RayOcc also managed to keep the number of 3D blobs (Gaussians) relatively low (around 19,251 per scene on average) while maintaining high accuracy, suggesting it doesn't just throw more data at the problem but uses it more intelligently.

Why This Matters

The core finding of this paper is that volumetric occupancy is a multi-label problem, not a single-choice problem. By stopping the computer from forcing objects to compete for a single "winner" status along a line of sight, RayOcc allows for a much more realistic representation of the world. It acknowledges that in a busy city, a single glance can reveal a car, a pedestrian, and a building all at once, and the computer should be allowed to believe in all of them simultaneously.

The authors conclude that this approach leads to "cleaner and more spatially consistent occupancy," particularly in challenging environments. While the method requires a moderate increase in computational cost compared to some older methods, the trade-off is a significantly more reliable understanding of the 3D world, which is crucial for the safety and planning of autonomous vehicles. The paper suggests that by aligning the mathematical model with the true nature of how light and objects interact (where multiple things can exist along one path), we can build smarter, safer eyes for our self-driving cars.

Technical Summary: RayOcc

Problem Statement

Camera-only 3D semantic occupancy prediction aims to infer voxel-wise scene semantics from multi-view images. However, this task faces fundamental challenges due to depth ambiguity and occlusion. Along a single camera ray, multiple spatially separated surfaces (e.g., a foreground vehicle, a thin fence behind it, and distant background structures) can project onto the same pixel. Consequently, ray-level occupancy supervision is inherently a multi-label existence problem, where multiple disjoint depth intervals along a single ray correspond to occupied regions.

Most existing camera-based occupancy methods inherit ray modeling strategies designed for depth estimation. These approaches typically discretize depth into bins and predict a softmax-normalized categorical distribution. This formulation enforces a conservation of probability mass, implicitly coupling depth intervals through competition. As a result, these methods favor a single dominant depth hypothesis per ray, often suppressing thin structures or fragmenting geometry under complex occlusion. This creates a formulation-level mismatch: volumetric occupancy supervision requires determining the independent existence of regions, not selecting a single competing outcome.

Methodology

The authors propose RayOcc, an occlusion-aware ray occupancy framework that reformulates ray modeling from depth selection to multi-label existence prediction. The approach consists of four main stages:

1. Ray Occupancy Modeling via Gaussian Mixture Intensity

Instead of predicting a categorical depth distribution, RayOcc estimates a non-normalized Gaussian mixture intensity along each ray.

Feature Extraction: Multi-view images are encoded using a Visual Geometry Grounded Transformer (VGGT) backbone to produce per-ray feature vectors, which are concatenated with Plücker coordinates to inject geometric context.
Mixture Intensity Network: For each ray, a network predicts parameters for $K$ Gaussian components (amplitude $a$ , mean $\mu$ , and standard deviation $\sigma$ ). The continuous intensity function $\lambda_r(d)$ is defined as the sum of these components. Crucially, the amplitudes are non-negative and not constrained to sum to 1, allowing multiple occupancy hypotheses to receive high intensity simultaneously.
Poisson Event Formulation: Occupancy within each depth bin is treated as an event occurrence. The integrated intensity of a bin is converted into an interval-wise existence probability using a Poisson formulation: $p = 1 - \exp(-\Lambda)$ . This removes the normalization constraint, allowing multiple depth bins to simultaneously have high occupancy probabilities without mutual competition.

2. Adaptive Component-Wise Sampling

The predicted mixture components serve as candidate occupancy hypotheses to initialize sparse 3D Gaussian primitives.

Activation Probability: The non-negative amplitude of each component is converted into an activation probability using a Poisson-like mapping ( $q = 1 - \exp(-\beta a)$ ).
Stochastic Sampling: During training, a binary gate is sampled from a Bernoulli distribution based on this probability. This allows each ray to instantiate a variable number of Gaussians, enabling multi-surface representation under depth ambiguity.
Initialization: Selected components are lifted to 3D positions using the ray origin, direction, and the component's mean depth. A lightweight grid-based subsampling operator then removes redundant Gaussians to form a compact latent set.

3. Refinement and Rasterization

The initialized Gaussian set is refined using a 3D sparse convolutional network that incorporates local 3D context. The network predicts residual updates for position, scale, rotation, opacity, and semantic features. Finally, the refined Gaussians are rasterized into a voxel-wise semantic occupancy grid using a probabilistic superposition operator.

4. Loss Function

The model is trained end-to-end with a weighted sum of two losses:

Occupancy Loss: A combination of weighted voxel-wise cross-entropy and Lovász-Softmax loss to handle class imbalance in the 3D grid.
Ray Distribution Loss: A combination of Binary Cross-Entropy (BCE) and Dice loss applied to the predicted per-ray bin occupancy probabilities, supervised by binary ground-truth labels derived from dense occupancy annotations.

Key Contributions

Mixture Intensity Network: A novel framework that predicts non-normalized Gaussian mixture intensity along rays and maps integrated intensity to interval-wise existence probabilities, explicitly modeling multi-label occupancy without enforcing mutual exclusivity.
Adaptive Component-Wise Sampling: A mechanism that converts mixture amplitudes into activation probabilities, enabling a variable number of Gaussian primitives to be initialized per ray, thereby supporting multi-hypothesis depth modeling.
State-of-the-Art Performance: The method achieves superior performance on the nuScenes benchmark compared to existing Gaussian-based occupancy methods.

Experimental Results

Experiments were conducted on the nuScenes benchmark using the dense 3D semantic occupancy annotations from SurroundOcc.

Overall Performance: RayOcc achieved a state-of-the-art IoU of 34.84 and mIoU of 22.03 among compared Gaussian-based methods. This represents an improvement over VG3T (34.06 IoU, 21.74 mIoU) and GaussianFormer-2 (30.56 IoU, 20.02 mIoU).
Initialization Quality: In ablation studies comparing Gaussian sampling strategies, RayOcc demonstrated significantly higher placement accuracy. The percentage of Gaussians placed inside occupied voxels increased from 28.85% (GaussianFormer-2) to 61.07%, while the average distance to the nearest occupied voxel center decreased from 1.24m to 0.70m.
Efficiency: Despite initializing more Gaussians per scene on average (19,251 vs. 13,661 for VG3T) to support multi-hypothesis modeling, RayOcc achieved lower latency (243ms vs. 223ms for VG3T) and competitive memory usage, demonstrating an effective trade-off between representational capacity and computational cost.
Qualitative Improvements: Visualizations indicate that RayOcc produces cleaner, more spatially consistent occupancy, particularly in occluded regions. It recovers objects (e.g., trucks) more coherently near thin structures (e.g., fences) and reduces spurious predictions compared to single-depth initialization methods.

Significance

The paper claims that RayOcc addresses a fundamental formulation-level issue in camera-only 3D occupancy prediction: the mismatch between single-depth ray lifting and the multi-label nature of volumetric occupancy. By reformulating ray modeling to allow multiple occupancy hypotheses to coexist without collapsing into a single dominant peak, RayOcc provides more reliable geometry predictions under depth ambiguity and occlusion. The method demonstrates that moving away from normalized categorical depth distributions toward non-normalized intensity modeling significantly improves both the initialization of sparse 3D primitives and the final semantic occupancy accuracy.