💻 computer science

S3KF: Spherical State-Space Kalman Filtering for Panoramic 3D Multi-Object Tracking

This paper presents S3KF, a panoramic 3D multi-object tracking framework that utilizes a spherical state-space Kalman filter on a unit sphere to effectively fuse quad-fisheye camera detections with rotating LiDAR data, achieving decimeter-level accuracy and robust identity continuity in wide-field industrial environments without requiring motion-capture infrastructure.

Original authors: Zhongyuan Liu, Shaonan Yu, Jianping Li, Pengfei Wan, Xinhang Xu, Pengfei Wang, Maggie Y. Gao, Lihua Xie

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhongyuan Liu, Shaonan Yu, Jianping Li, Pengfei Wan, Xinhang Xu, Pengfei Wang, Maggie Y. Gao, Lihua Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to keep track of a group of friends running around in a giant, open field. You have two tools to help you: a fisheye camera (which sees everything around you in a distorted, 360-degree circle) and a spinning laser scanner (LiDAR) that measures exactly how far away things are.

The problem? Tracking people in a 360-degree view is like trying to draw a map of the entire Earth on a flat piece of paper. If you try to flatten the world, the poles get stretched and torn, and things near the top and bottom look weirdly distorted. Traditional tracking software tries to draw the map on a flat "image paper," which causes it to get confused when people move near the edges or get blocked by obstacles.

This paper introduces S3KF, a new way of tracking that solves this by changing the "map" itself.

The Core Idea: The "Globe" Instead of the "Map"

Instead of trying to flatten the world onto a 2D screen (like a standard photo), the authors decided to keep the world as a globe.

The Old Way (Flat Map): Imagine trying to track a runner on a flat map. When they run near the North Pole, the map stretches them out, making it hard to know where they really are. If they disappear behind a tree, the flat map gets confused about where they reappear.
The S3KF Way (The Globe): Imagine your tracker is a smart robot standing in the center of a giant, invisible sphere. Instead of looking at a flat picture, it looks at the surface of the sphere.
- When a person moves, the robot just updates their position on the curve of the sphere.
- There are no "edges" or "poles" where the map breaks. It's a smooth, continuous circle.
- To do the math, the robot uses a tiny, flat "tangent plane" (like a small piece of paper touching the sphere at one point) to do calculations, but it constantly rolls that paper along the sphere as the person moves. This keeps the math simple but the geometry perfect.

The Team: Eyes and Lasers

The system uses two sensors working together, like a detective with two different senses:

The Fisheye Camera (The Eyes): It sees what the object is (a person, a bike) and how big it looks. But it's bad at knowing exactly how far away they are, especially in a distorted 360-degree view.
The Rotating LiDAR (The Laser Sense): It spins around like a lighthouse, sending out laser beams to measure the exact distance to everything. It knows the 3D shape of the world perfectly but doesn't "see" colors or details.

The Magic Fusion:
S3KF combines these two. It takes the "what" from the camera and the "how far" from the laser. Because both are mapped onto the same "globe" (the sphere), they fit together perfectly. The laser tells the camera, "That blurry blob is actually 5 meters away," and the camera tells the laser, "That dot is a person, not a tree."

The "Ground Truth" Hack: The Invisible String

One of the hardest parts of testing a tracking system is knowing the real answer to see if your system is right. Usually, you need a massive, expensive studio with infrared cameras everywhere (like a motion-capture stage).

The authors came up with a clever, low-cost solution:

They gave each person a small backpack with a tiny laser scanner and a Wi-Fi chip.
They built a "master map" of the area using a big scanner beforehand.
As the people walked around, their backpacks compared what they saw to the "master map" to figure out exactly where they were.
The Analogy: It's like giving your friends a GPS that doesn't need satellites, but instead recognizes the unique "fingerprint" of the buildings and trees around them to know their location. This gave the researchers a perfect "answer key" to grade their tracking system without needing a $1 million studio.

Why Does This Matter?

The results were impressive. The system could track people running, hiding behind obstacles, and moving in circles, even on a robot dog or a flying drone.

Fewer "Identity Swaps": In old systems, if two people crossed paths, the computer often got confused and swapped their names (e.g., "That's Bob now, not Alice"). S3KF kept the names correct almost all the time because the "globe" math didn't get confused by the distortion.
Real-Time: It runs fast enough to be used on a robot right now, not just on a supercomputer.

Summary

S3KF is like upgrading from a flat, distorted map of the world to a smooth, rotating globe. By treating the world as a sphere and combining camera eyes with laser depth, it can track people in a 360-degree world without getting lost, confused, or swapping identities. It's a smarter way for robots to "see" the whole world at once.

1. Problem Statement

Panoramic 3D multi-object tracking (MOT) is critical for industrial safety, wide-area robotics, and infrastructure-light deployments. However, existing solutions face significant challenges:

Image-Plane Limitations: Traditional trackers (e.g., ByteTrack) operate in the 2D image plane. In panoramic/fisheye imagery, these suffer from severe non-uniform projection distortions and singularities near the poles (e.g., in equirectangular projections), leading to unstable state estimation.
Euclidean 3D Redundancy: Standard 3D trackers often represent object direction using 3D Euclidean vectors with unit-norm constraints. This introduces redundant degrees of freedom, complicating filtering and failing to naturally unify angular, scale, and depth estimation.
Sensor Gaps: Camera-only systems lack metric depth, making association fragile during occlusion or scale changes. Fixed LiDARs have blind spots, while simply fusing sensors without a unified geometric state representation fails to resolve the core representation problem.
Evaluation Bottleneck: Obtaining accurate ground truth for 3D human tracking in large, unstructured environments without expensive motion-capture infrastructure is difficult.

2. Methodology: S3KF Framework

The authors propose S3KF, a unified framework integrating a motorized rotating LiDAR and a quad-fisheye camera rig. The core innovation is a geometry-consistent state representation on the unit sphere ( $S^2$ ).

A. Spherical State Representation

Instead of tracking in the distorted image plane or redundant 3D space, the system models the object's bearing vector $g_k$ on the unit sphere $S^2$ .

Tangent-Plane Parameterization: To perform linear filtering on the nonlinear manifold $S^2$ , the system uses a local tangent plane $T_{\bar{g}_k}S^2$ at a reference direction $\bar{g}_k$ .
State Vector ( $x_k$ ): A 10-dimensional vector is defined:
$x_k = [w_{1,k}, w_{2,k}, \dot{w}_{1,k}, \dot{w}_{2,k}, \ell_k, \dot{\ell}_k, h_k, \dot{h}_k, d_k, \dot{d}_k]^\top$
Where:
- $(w_1, w_2)$ : 2D tangent-plane coordinates representing the bearing direction.
- $(\dot{w}_1, \dot{w}_2)$ : Angular velocities.
- $(\ell, h)$ : Bounding box aspect ratio and height (observable from images).
- $(d)$ : Metric depth (observable from LiDAR).
Log/Exp Maps: The system utilizes Riemannian logarithmic and exponential maps to convert between global spherical coordinates and local Euclidean tangent coordinates for Kalman filtering operations.

B. Extended Spherical Kalman Filter

The framework employs an Extended Kalman Filter (EKF) tailored for spherical geometry:

Motion Model: Assumes constant velocity dynamics for all state components (position, scale, depth).
Measurement Fusion:
- Direction: Projects panoramic camera detections onto the tangent plane using the log-map.
- Scale & Depth: Fuses bounding box dimensions from the camera and metric depth from the rotating LiDAR.
- Asynchronous Updates: Handles asynchronous sensor data by performing sequential updates (Image update $\to$ LiDAR update) or joint updates if synchronized.
Data Association: Uses a Mahalanobis distance gating mechanism combined with spherical Intersection-over-Union (IoU) and depth consistency checks, followed by the Hungarian algorithm.

C. Ground-Truth Generation Pipeline

To enable quantitative evaluation without motion-capture systems, the authors developed a map-based ground-truth pipeline:

Hardware: Wearable devices equipped with LiDAR and WiFi.
Process: A pre-built global LiDAR map is created. Wearable devices perform online localization by registering real-time scans to this global map.
Result: This generates globally consistent 6-DoF trajectories for multiple subjects with centimeter-level accuracy (planar error < 3 cm), independent of external infrastructure.

3. Key Contributions

Hardware Platform: Integration of a motorized rotating LiDAR with a quad-fisheye camera rig for synchronized, full-surround geometric and appearance sensing.
Spherical State Formulation: A novel geometry-consistent state representation on $S^2$ using a 2-DOF tangent-plane parameterization. This eliminates image-space singularities and Euclidean redundancy while unifying 2D and 3D tracking.
Extended Spherical Kalman Filter: A filtering pipeline that fuses panoramic 2D detections and LiDAR depth within a single state-space model, preserving spherical geometry during inference.
Infrastructure-Light Evaluation: A map-based ground-truth generation system using wearable LiDAR localization, enabling rigorous quantitative evaluation in large-scale real-world environments.

4. Experimental Results

Experiments were conducted on a NVIDIA Jetson AGX Orin platform in real-world outdoor and indoor scenarios involving multiple moving participants.

Tracking Accuracy: The system achieved decimeter-level planar tracking accuracy.
- Overall Mean RMSE (x-y plane): 0.187 m.
- Performance remained robust across static, dynamic, and high-speed motion scenarios, as well as in the presence of frequent occlusions.
Identity Continuity: Compared to the 2D baseline (ByteTrack), S3KF significantly reduced identity switches (ID switches):
- ByteTrack: 98 total ID switches across four test sequences.
- S3KF: 16 total ID switches.
- In specific sequences (Ground-Outdoor, Ground-Indoor, Aerial-Outdoor), ID switches dropped from ~26-27 to just 3.
Platform Agnosticism: The framework demonstrated robustness when deployed on both quadruped robots (ground) and UAVs (aerial), handling ego-motion disturbances, rapid viewpoint changes, and vibration without tracking loss.
Real-Time Performance: The system operates in real-time (30 Hz filter update) on embedded hardware.

5. Significance

Theoretical Advancement: S3KF resolves the fundamental geometric mismatch in panoramic tracking by moving the state space from the distorted image plane to the intrinsic unit sphere, avoiding singularities and redundancy.
Practical Impact: It provides a viable solution for industrial-scale safety monitoring and wide-area robotics where full 360° coverage and metric accuracy are required without the cost of motion-capture infrastructure.
Scalability: The map-based ground-truth method allows for scalable evaluation of multi-person tracking in large, unstructured environments, a capability previously limited to controlled lab settings.
Unified Framework: By treating direction, scale, and depth as a unified state, the framework offers a flexible interface that can seamlessly transition between 2D panoramic tracking and metric 3D tracking depending on sensor availability.