💻 computer science

Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation

This paper proposes a framework that extends existing online 2D multi-camera tracking systems to 3D space by leveraging depth-based point-cloud reconstruction and an enhanced data association mechanism, achieving third place on the 2025 AI City Challenge 3D MTMC leaderboard.

Original authors: Vu-Minh Le, Thao-Anh Tran, Duc Huy Do, Xuan Canh Do, Huong Ninh, Hai Tran

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vu-Minh Le, Thao-Anh Tran, Duc Huy Do, Xuan Canh Do, Huong Ninh, Hai Tran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to keep track of a busy crowd of people walking through a large building, but you have a dozen security cameras watching from different angles. Your goal is to know exactly where everyone is in 3D space (not just on a flat screen) and to make sure you don't lose track of "Person A" just because they walked behind a pillar or switched from Camera 1's view to Camera 2's view.

This paper presents a new "smart assistant" for security systems that solves this problem without needing to completely rebuild the existing software. Here is how it works, broken down into simple steps:

1. The Problem: "Retrofitting" is Hard

Most old security systems are great at tracking things on a flat, 2D screen (like a video game). But to know where something is in the real 3D world, you usually have to throw away the old system and build a new one from scratch. That is expensive and difficult.

The authors wanted a way to take a working 2D system and "upgrade" it to 3D without tearing it down.

2. The Solution: A Two-Stage Assembly Line

Think of their system as a factory with two main stations:

Station 1: The 2D Detective (Tracking)
First, the system uses the existing cameras to find people and objects. It draws a box around them on the 2D screen and gives them a temporary ID (like a name tag).

The "Name Tag" Trick: The authors invented a clever way to make sure the name tags stay consistent. If a person is seen by Camera A, B, and C, the system checks if the "local" IDs from those cameras match up with what was seen in the previous second.
The "Split" Mechanism: Sometimes, two people might look so similar or get so close that the system accidentally thinks they are one person. The authors' system acts like a detective who realizes, "Wait, that's actually two different people!" and splits the group back into two separate tracks.

Station 2: The 3D Sculptor (Depth Aggregation)
Once the system knows who is who in 2D, it moves to the second stage to figure out where they are in 3D.

Gathering Clay: The system takes the 2D boxes and combines them with "depth maps" (which are like invisible 3D scanners that tell the camera how far away things are). It uses this to build a rough "cloud of dots" (a point cloud) for each person, essentially sculpting them out of digital dust.
Cleaning Up: Because the cameras aren't perfect, this cloud of dots can be messy or have holes (like when someone is partially hidden). The system uses a "noise filter" to smooth out the cloud and remove the stray dots.
Fitting the Box: Once the cloud is clean, the system fits a perfect 3D cardboard box around it.
The "Fusion" Step: Sometimes, the system might accidentally create two boxes for the same person because of a glitch. The authors' method acts like a glue gun, merging those duplicate boxes into one single, accurate 3D box.
The "Yaw" Refinement: Finally, to make sure the box is facing the right way (e.g., if a person is walking sideways vs. forward), the system looks at where the person was 10 seconds ago and where they are now. It calculates the direction they are moving and rotates the 3D box to match that direction.

3. The Result: A Real-World Test

The team tested this "upgrade kit" on a massive dataset called the 2025 AI City Challenge, which simulates complex environments like warehouses and hospitals with up to 50 cameras.

The Outcome: Their method didn't just work; it was highly effective. They took an existing 2D tracking system and, by adding their "3D Sculptor" and "Name Tag" upgrades, they achieved 3rd place in the global competition.
Why it matters: They proved you don't need to scrap your old 2D security software to get high-quality 3D tracking. You just need to add this specific "late aggregation" layer on top of it.

In a nutshell: They figured out how to take a flat, 2D video tracking system and give it "depth perception" by combining camera views, cleaning up the digital data, and smartly managing ID tags, all without needing to rebuild the whole system from the ground up.

Technical Summary: Online 3D Multi-Camera Perception through Robust 2D Tracking and Depth-based Late Aggregation

Problem Statement

Multi-Target Multi-Camera Tracking (MTMC) is a critical computer vision task for automating large-scale surveillance. While 2D MTMC has achieved remarkable performance, extending these capabilities to 3D space remains challenging. Existing 3D MTMC approaches often require fusing multi-view data or detections into a unified 3D space before tracking (e.g., EarlyBird, BEV-SUSHI). This "early fusion" approach typically necessitates replacing all 2D tracking components from the ground up, which is often impractical for existing large-scale surveillance systems due to the complexity and cost of such overhauls. Furthermore, 3D tracking must effectively handle multi-view information fusion, partial occlusions, and scalable object-level 3D geometry representation.

Methodology

The authors propose a novel framework that extends any existing online 2D multi-camera tracking system into 3D space through Late 3D Bounding Box Aggregation. The pipeline operates in two primary phases:

1. Enhanced Online 2D Multi-Camera Tracking

The first phase focuses on obtaining consistent 2D bounding boxes and global identities across views.

Detection & Feature Extraction: The system uses a Transformer-based Co-DETR detector. For feature extraction, it employs CLIP-ReID for appearance embeddings and RTMPose for pose estimation (14 keypoints) to identify foot points.
Spatial Data Association: At each timestep, detected 2D targets are clustered based on ReID appearance features (cosine distance) and top-down location (Euclidean distance via homography). Agglomerative clustering groups targets into unique entities.
Temporal Data Association (MOT ID Consistency): To assign global IDs, the authors introduce an MOT ID Consistency mechanism. This approach leverages the consistency of local MOT IDs across consecutive frames.
- Consistency Matching: A cluster is matched to an existing track if its group of local IDs overlaps significantly with the previous track's local IDs.
- Track Splitting: If a cluster contains "unexpected" local IDs (IDs not seen in the previous track) that are more similar to the expected track than the old local ID, the system splits the track. This prevents ID switching caused by occlusions or noisy features.
- State Management: Tracks are managed in three states: Tentative, Confirmed, and Lost, with specific rules for reactivation and new track creation.

2. Late 3D Bounding Box Aggregation

Once 2D tracking and global IDs are established, the system reconstructs 3D representations without modifying the 2D tracker.

Depth to Point Cloud: For each tracked object, the system segments the object within its 2D bounding box using SAM2. It extracts depth values from the provided depth maps for pixels within the mask. These 2D pixels are projected into 3D camera coordinates and then transformed into a global world coordinate system using extrinsic calibration parameters.
Point Cloud to 3D Bounding Box: The resulting point clouds often contain noise and fragmentation due to occlusion or projection errors.
- Clustering: DBSCAN is applied to remove noise and isolate the object's point cloud.
- Box Estimation: Dimensions are estimated using percentiles of the point coordinates. To handle cases where the point cloud is too sparse or noisy, the system validates the predicted volume against the mean volume of the class in the training set, clamping dimensions if they fall outside a defined range ( $\alpha_{lower}$ to $\alpha_{upper}$ ).
3D Bounding Box Fusion: Due to potential errors in 2D spatial association, a single object may be assigned multiple 3D boxes with different IDs. The system performs a weighted fusion of overlapping boxes (using 3D IoA) to merge them into a single, accurate 3D bounding box.
Yaw Refinement: To improve orientation accuracy, the yaw angle is refined based on the target's movement trajectory. The angle is calculated from the displacement vector between frames $t-10$ and $t$ , updating only if the movement exceeds a specific threshold.

Key Contributions

Flexible Framework: A method to extend traditional online 2D MTMC systems to output 3D tracking results via late aggregation, avoiding the need to rebuild 2D components.
Enhanced Association Mechanism: An online association strategy using MOT ID Consistency to propagate global IDs across frames by leveraging local ID consistency, improving cross-camera tracking performance.
3D Reconstruction Mechanism: A pipeline that extends ID-assigned 2D bounding boxes into 3D space using segmentation, depth map extraction, point-cloud clustering, and geometric refinement to create 3D bounding boxes.
Validation: The method was validated on the 2025 AI City Challenge 3D MTMC dataset, achieving third place on the public leaderboard.

Results

The proposed system was evaluated on the AIC25 MTMC dataset (23 synthetic scenes, up to 50 cameras).

2D Performance: On the validation set, the MOT ID Consistency matching improved the HOTA score by 20.46% (from 48.63 to 69.09), primarily driven by a 29.03% increase in Association Accuracy (AssA).
3D Performance: On the public test set, the full pipeline achieved a HOTA score of 28.75%.
- The MOT ID Consistency module contributed to a ~5% increase in Detection Accuracy (DetA) and a 7% increase in AssA.
- The Late 3D Aggregation module provided the most significant boost, increasing DetA by 13% and HOTA by 10%.
- Yaw Refinement further improved Localization Accuracy (LocA), pushing the final HOTA to 28.75%.
Leaderboard Standing: The system secured 3rd place in Track 1 of the 2025 AI City Challenge, outperforming other teams such as UTE AI Lab (25.39 HOTA) and SKKU-AutoLab (63.13 HOTA in a different context, though the paper lists TeamQDT at 28.75 HOTA for the specific 3D track).

Significance and Claims

The paper claims that its primary significance lies in offering a practical and cost-effective solution for upgrading existing surveillance infrastructure. By utilizing "late aggregation," the authors demonstrate that high-quality 3D perception can be achieved without the prohibitive cost and complexity of replacing established 2D tracking components. The framework effectively bridges the gap between 2D tracking and 3D localization, leveraging depth information and robust ID consistency to handle occlusions and multi-view challenges, as evidenced by its competitive performance in a rigorous benchmark.