Geometry Beats Estimated Depth: RGB-Only Multi-Camera 3D Tracking under Sim2Real

Imagine you are trying to build a 3D movie of a busy warehouse using only flat, 2D pictures taken from security cameras. This is the world of multi-camera 3D perception, a field where computers try to understand the real, three-dimensional world just by looking at flat images. Usually, to do this perfectly, robots use special sensors that measure distance directly, like a bat using sonar or a human using two eyes to judge depth. But in this specific challenge, the robots are "blind" to distance; they only have standard RGB cameras (the kind on your phone) and a set of rules about how the cameras are positioned. The big question scientists are asking is: If you can't measure distance directly, what's the best way to guess it? Should you try to use a super-smart AI to guess the depth of every single pixel in the image (like a psychic guessing the distance to a tree), or should you rely on simple, hard geometry—using the known angles of the cameras to triangulate where things are on the floor?

This paper, written by a team from LSU New Orleans and PinPark for the AI City Challenge 2026, dives right into that debate. They set up a "tug-of-war" between two very different strategies for tracking people and robots in a warehouse. One team (the authors) bet on geometry: using the known map of the floor and the camera angles to lift 2D shapes into 3D space. The other strategy, which they tested as a baseline, bet on estimated depth: using a fancy AI to guess how far away every pixel is, turning the flat images into a 3D cloud of points (like a digital point-cloud map) and then running a 3D detector on it. The results were shocking and clear. The geometry team won by a landslide, while the "guess-the-depth" team completely collapsed. The paper suggests that in this specific "Sim2Real" setting (where the robot is trained on fake data but tested on real data), having a consistent, shared understanding of the floor across all cameras is far more important than having a perfect, pixel-by-pixel guess of how deep an object is.

The Great Warehouse Heist: Two Ways to See in 3D

The story begins in a giant, simulated warehouse that is about to become very real. The challenge is to track moving objects—like forklifts, pallet trucks, and even humanoids—using only video feeds from multiple cameras. The catch? The cameras don't have depth sensors. They are just regular eyes. The AI has to figure out where everything is in 3D space (up/down, left/right, forward/back) just by looking at flat pictures.

The authors set up two different "detectives" to solve this case.

Detective A: The Geometry-First Team
This team took a very logical, "old-school" approach. They didn't try to guess the depth of every single pixel. Instead, they said, "We know the floor is flat, and we know exactly where the cameras are pointing."

1. Spot the Object: First, they used a powerful detector (called YOLO11x) to find 2D boxes around objects in each camera view.
2. Lift to the World: They took the bottom-center of those 2D boxes and projected them onto the known floor plane using math (homography). It's like shining a flashlight from the camera through the bottom of the box onto the floor to see where it lands.
3. Fuse and Track: Since the same forklift might be seen by three different cameras, they merged these "lifted" points into a single 3D location. Then, they tracked the object as it moved across the warehouse floor.
4. The Secret Sauce: They added a final step called "offline stitching." Imagine a detective who realizes, "Wait, I lost that suspect for five seconds, but they reappeared right here. It's the same person!" They linked the broken pieces of the track together to fix the identity.

Detective B: The Pseudo-LiDAR Team
This team tried the "modern" approach that had worked in the past when depth data was available. They tried to fake it.

1. Guess the Depth: They used a fancy AI model (like D4RT or Metric3D) to look at the flat image and guess the distance to every single pixel. This turns the 2D image into a 3D cloud of points, mimicking a real 3D sensor (LiDAR).
2. Detect in 3D: They fed this "fake" 3D cloud into a 3D detector (V-DETR) to find the objects.
3. The Hope: They hoped that even without real depth, the AI could guess it well enough to build a perfect 3D map.

The Shocking Result: Geometry Wins, Guessing Loses

The results were not just a win; they were a landslide. The Geometry-First team achieved a score of 13.0 HOTA (a measure of how well they tracked and located objects). The Pseudo-LiDAR team, the "guessing" team, collapsed to a score of 0.12. That is roughly a hundred times worse.

Why did the guessing team fail so badly? The paper suggests it wasn't because their AI was bad at guessing depth on a single image. The problem was consistency.
Imagine you and three friends are trying to draw a map of a room based on photos you each took.

• The Geometry Team all agreed on where the floor was. Even if they weren't perfect, they all agreed on the ground plane. When they combined their maps, the floor was flat, and the forklifts stood upright.
• The Pseudo-LiDAR Team each guessed the depth independently. One friend thought the floor was 1 meter high, another thought it was 2 meters, and a third thought it was floating in the air. When they tried to combine their maps, the floor became a warped, jagged mess. The forklifts ended up floating in the sky or buried underground.

The authors call this "cross-view inconsistency." Even if the AI guesses the depth of a single pixel correctly, if it guesses differently for the same floor in different camera views, the 3D map breaks. The paper shows that for this specific task, having a shared, consistent geometry (the floor plan) is much more important than having a perfect, pixel-by-pixel depth guess.

What Didn't Work (And Why)

The authors didn't just stop at comparing the two teams; they tried to fix the losing team and improve the winning team with every trick in the book. They tested:

• Better Detectors: They tried swapping the main detector for newer, fancier models (like RT-DETR) or combining them. Result: No help. The problem wasn't the detector; it was the lack of real-world data to train on.
• Sliced Detection: They tried breaking the images into tiny pieces to catch small objects (like a pallet truck far away). Result: This actually made things worse. It found more objects, but most were false alarms (ghosts), confusing the tracker.
• Learning to Lift: They tried to replace the simple math of "lifting" the box to the floor with a neural network that "learned" how to do it. Result: Disaster. The score dropped to near zero. The simple math was actually more reliable than the learning.
• Appearance Re-ID: They tried to identify objects by how they looked (e.g., "that's the red forklift"). Result: It didn't help because the lighting and angles changed too much in the real world.

The only thing that helped the Geometry team was offline stitching. By going back after the fact and fixing the broken tracks (linking the "lost" and "found" pieces of the same object), they improved their score slightly. This tells us that the main bottleneck is detection quality (finding the object in the first place), not the tracking logic.

The Big Takeaway

The paper concludes with a clear lesson for anyone building robots that need to see in 3D without special sensors: Don't try to guess the depth if you can use geometry.

In a world where the robot is trained on fake data but tested on the real world, trying to learn the depth of every pixel leads to a chaotic, inconsistent map. Instead, sticking to the known rules of the world (the floor is flat, the cameras are fixed) and using simple, explicit math to lift 2D detections into 3D space is far more reliable. The "magic" of guessing depth isn't magic enough to overcome the confusion of the real world. The best path forward isn't better depth-guessing AI; it's better detectors that can actually see the objects clearly in the first place.