OccTrack360: 4D Panoptic Occupancy Tracking from Surround-View Fisheye Cameras

This paper introduces OccTrack360, a new benchmark for 4D panoptic occupancy tracking from surround-view fisheye cameras featuring long, diverse sequences and principled voxel visibility annotations, alongside the proposed Focus on Sphere Occ (FoSOcc) framework that effectively addresses fisheye distortion and localization challenges to establish a strong baseline for future research.

The Gist

Imagine you are driving a car, but instead of looking through a standard windshield, you have six giant, fish-eye lenses wrapped around your vehicle, giving you a 360-degree view of the world. This is how many modern self-driving cars "see."

However, there's a big problem: Fish-eye lenses are weird. They stretch and warp the image, making straight lines look curved and objects near the edges look huge or tiny depending on where they are. Most current self-driving software was trained on "normal" camera images (like a human eye), so when they try to use fish-eye data, they get confused. They struggle to figure out exactly where things are in 3D space, especially when tracking moving objects over time.

This paper, "OccTrack360," solves two major problems:

1. The Data Gap: There was no good "test drive" (benchmark) for this specific type of fish-eye, 360-degree tracking.
2. The Software Glitch: The existing software couldn't handle the warped images well.

Here is the breakdown of their solution using simple analogies.

---

1. The New Test Track: OccTrack360

Think of training a self-driving AI like training a race car driver. You need a test track that mimics real life.

• The Old Tracks: Previous tests were like short, straight tracks with narrow views (pinhole cameras). They didn't test how a car handles long, winding roads or seeing everything around it at once.
• The New Track (OccTrack360): The authors built a massive, new digital test track.
  • Longer Sequences: Instead of a 10-second clip, they have videos lasting minutes (up to 2,000+ frames), testing if the car can remember where a pedestrian was 5 minutes ago.
  • 360-Degree Vision: It uses data from all the fish-eye cameras, not just the front.
  • The "Invisible Wall" Map: They created a special map that tells the AI exactly what parts of the 3D world are actually visible through the fish-eye lens and what parts are blocked by the car itself or other objects. This prevents the AI from guessing about things it can't possibly see.

2. The New Driver: FoSOcc (Focus on Sphere Occ)

Now that they have a better test track, they built a new "driver" (AI model) called FoSOcc to run on it. This driver has two special superpowers to handle the fish-eye distortion.

Superpower A: The "Center-Focus" Glasses (Center Focusing Module)

The Problem: In a warped fish-eye image, the edges are blurry and stretched. If the AI tries to guess the exact edge of a car or a pole, it might get it wrong because the distortion makes the edge look fuzzy.
The Analogy: Imagine trying to draw a circle on a piece of rubber that is being stretched. The edges are hard to pin down. But the center of the circle stays relatively stable.
The Solution: Instead of obsessing over the shaky edges, the AI puts on "Center-Focus Glasses." It ignores the messy edges and focuses intensely on the center point of every object. By anchoring its understanding to the stable center, it can figure out where the object is, even if the edges are warped. This makes tracking small objects (like a traffic cone) much more accurate.

Superpower B: The "Un-Warping" Lens (Spherical Lift Module)

The Problem: Standard 3D mapping assumes the world is flat (like a piece of paper). But fish-eye cameras see the world on a sphere (like a globe). Trying to map a sphere onto a flat piece of paper without distortion is impossible (think of how a flat map of the world distorts Greenland).
The Analogy: Imagine trying to flatten an orange peel without tearing it. You can't do it perfectly. Standard software tries to force the orange peel flat, which breaks the geometry.
The Solution: The authors built a "Spherical Lift." Instead of forcing the image to be flat, they let the AI understand that the world is curved. They use a mathematical model (the Unified Projection Model) that treats the camera's view as a globe. This allows the AI to "lift" 2D pixels into 3D space correctly, respecting the curve of the fish-eye lens. It's like realizing you are looking at a globe, not a flat map, so the distances make sense again.

3. The Results

When they tested this new system:

• On Standard Data: It got better at recognizing things like traffic signs and general objects, proving that focusing on the "center" helps even with normal cameras.
• On the New Fish-Eye Data: It became the new gold standard. It could track objects through the warped, 360-degree view much better than previous methods.

Summary

In short, the authors realized that fish-eye cameras are great for self-driving cars but terrible for current software.

1. They built a new, harder test track (OccTrack360) that actually uses fish-eye data.
2. They built a new AI driver (FoSOcc) that knows how to handle warped images by:
   • Focusing on the stable centers of objects (ignoring the messy edges).
   • Understanding that the view is curved like a sphere, not flat like a photo.

This helps self-driving cars see the whole world clearly, without getting dizzy from the fish-eye distortion.

Technical Summary

Here is a detailed technical summary of the paper "OccTrack360: 4D Panoptic Occupancy Tracking from Surround-View Fisheye Cameras."

1. Problem Statement

The paper addresses a critical gap in autonomous driving and robotics perception: 4D Panoptic Occupancy Tracking (simultaneously tracking geometry, semantics, and instance identities over time) using surround-view fisheye cameras.

Current state-of-the-art methods and benchmarks face three main limitations:

• Lack of Fisheye Support: Existing benchmarks (e.g., Occ3D-Waymo, Occ3D-nuScenes) primarily rely on pinhole cameras with limited Fields of View (FoV), failing to capture the full 360° environment essential for close-proximity safety.
• Insufficient Temporal Depth: Existing datasets often feature short sequences, making it difficult to evaluate long-horizon tracking and temporal consistency.
• Inadequate Supervision for Fisheyes: Standard visibility constraints (occlusion masks) and projection models assume pinhole geometry. They fail to account for severe radial distortion, spherical projection, and the specific visibility constraints of fisheye lenses, leading to inaccurate 2D-to-3D lifting and poor instance localization.

2. Methodology

The authors propose a two-pronged solution: a new benchmark (OccTrack360) and a novel framework (FoSOcc).

A. OccTrack360 Benchmark

Built upon the KITTI-360 dataset, OccTrack360 introduces a comprehensive pipeline for generating 4D panoptic labels from fisheye data:

• Data Scale: Provides significantly longer and more diverse sequences (174–2,234 frames) compared to prior benchmarks.
• Instance-Level Annotations: Offers voxel-level ground truth with instance IDs for both dynamic agents and static structures (e.g., poles, buildings).
• Principled Visibility Masks:
  • All-Direction Occlusion Mask: Unlike previous methods that only mask rays intersecting occupied voxels, this mask covers all directions in the voxel domain, preventing the model from learning on invalid "blind" directions.
  • MEI-based Fisheye FoV Mask: Derived using the Unified Projection Model (MEI), this mask explicitly defines the valid field of view for fisheye cameras, ensuring voxels outside the physical lens range are not penalized.

B. FoSOcc (Focus on Sphere Occ) Framework

To handle the unique challenges of fisheye imagery, the authors propose FoSOcc, which integrates two key modules:

1. Center Focusing Module (CFM):

   • Problem: Standard boundary-based offsets are sensitive to depth noise and fisheye distortion, causing unstable gradients.
   • Solution: Shifts supervision from volatile object boundaries to stable instance centers. It aggregates offsets from all six cardinal directions to create a "center-peaked" product feature ($\delta_p$).
   • Mechanism: Uses instance-level normalization to make the geometric cues scale-invariant. This acts as a soft Gaussian constraint, anchoring the model's attention on the object center, thereby improving localization robustness even in distorted peripheral regions.

2. Spherical Lift Module (SLM) / Fisheye-based Enhanced Lifting (FEL):

   • Problem: Standard Lift-Splat-Shoot (LSS) pipelines assume a pinhole camera model, which fails under the non-linear radial distortion of fisheye lenses.
   • Solution: Extends the lifting operation to fisheye imaging using the Unified Projection Model (UCM).
   • Mechanism: Explicitly incorporates the mirror parameter ($\xi$) and distortion coefficients to map 2D image features onto a displaced unit sphere. It rectifies the radial distance to ensure geometric validity before projecting points into 3D voxel space, enabling accurate 2D-to-3D feature lifting under wide FoV.

3. Key Contributions

1. OccTrack360 Benchmark: The first benchmark dedicated to 4D panoptic occupancy tracking from surround-view fisheye cameras, featuring long sequences, instance-level voxel annotations, and specialized visibility masks (all-direction occlusion and MEI-based FoV).
2. FoSOcc Framework: A fisheye-oriented tracking architecture that solves distorted projection and localization issues via the Center Focusing Module (CFM) and Spherical Lift Module (SLM).
3. Novel Supervision Signals: Introduction of all-direction occlusion masks and MEI-based FoV masks to provide faithful supervision for wide-FoV voxel reasoning.
4. State-of-the-Art Performance: Demonstrated significant improvements over existing baselines on both standard (Occ3D-Waymo) and new (OccTrack360) benchmarks.

4. Experimental Results

The method was evaluated on Occ3D-Waymo and OccTrack360 using metrics like Occupancy Segmentation Quality (OccSQ), Association Quality (OccAQ), and Tracking Quality (OccSTQ).

• On Occ3D-Waymo:
  • FoSOcc achieved relative gains of 11.1% on traffic signs and 20.7% on general objects compared to the baseline.
  • Showed a 26.1% improvement in Association Quality (AQ) for cyclists, highlighting its ability to track small, dynamic objects.
• On OccTrack360:
  • Segementation: Improved Overall OccSQ from 12.90 to 13.54 (all FoV) and 14.49 (Fisheye only).
  • Specific Gains: Massive improvements in challenging categories like "Parking" (0 $\to$ 6.47) and "Other Structure" (7.02 $\to$ 12.49).
  • Trade-off: While segmentation improved, tracking metrics (OccSTQ/OccAQ) showed a slight decrease compared to the baseline, indicating that temporal association in fisheye sequences remains a challenging area for future work.
• Ablation Studies: Confirmed that both the instance-level normalization and the supervised focus feature in CFM are critical. Combining them yields the best performance, particularly for small-scale objects.

5. Significance and Future Work

• Bridging the Gap: This work is pivotal for moving autonomous perception from restricted pinhole views to comprehensive 360° fisheye understanding, which is crucial for safety in dense urban environments.
• Geometric Rigor: By formally integrating the Unified Projection Model into occupancy lifting, the paper provides a mathematically sound approach to handling fisheye distortion, moving beyond ad-hoc data augmentation.
• Foundation for Research: OccTrack360 establishes a necessary baseline for future research in 4D occupancy tracking, encouraging the development of models that can handle long-term, surround-view dynamics.
• Future Directions: The authors plan to integrate specialized 2D backbones designed for fisheye geometry and investigate specialized depth estimation techniques to further mitigate non-linear distortion effects.

In summary, OccTrack360 and FoSOcc represent a significant step forward in making 4D panoptic occupancy tracking viable for real-world autonomous systems equipped with surround-view fisheye cameras.