💻 computer science

EgoTrack3D: A Modular Framework for Egocentric 3D Object Tracking

EgoTrack3D is a modular framework that reconstructs and maintains dynamic 3D scene representations from egocentric RGB video by lifting 2D masks into a global coordinate frame with motion scoring and voxel-based merging, achieving state-of-the-art tracking accuracy on the ADT dataset while remaining robust under sparse and noisy observation conditions.

Original authors: Jan Kulik, Bjarni Dagur Thor Karason, Yung-Hsu Yang, Boyang Sun, Marc Pollefeys, Xi Wang

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jan Kulik, Bjarni Dagur Thor Karason, Yung-Hsu Yang, Boyang Sun, Marc Pollefeys, Xi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a pair of high-tech glasses that let you see the world in 3D, just like a robot or a video game character. To navigate a room safely, these glasses need to build a mental map of everything around them: where the chair is, where the coffee cup sits, and how a person walking by changes the scene. This is the world of 3D perception, a field where computers try to understand the physical world from camera videos.

Usually, computers are great at spotting things in flat, 2D pictures (like finding a cat in a photo). But when you move your head, the world shifts, objects hide behind each other, and things get tricky. To build a true 3D map, computers need to know not just what an object is, but exactly where it is in space and how it moves over time. This is called 3D object tracking. It's the difference between taking a single snapshot of a messy room and having a live, 3D model that updates every second as you walk through it. This is crucial for robots that need to avoid tripping over toys, or for augmented reality apps that want to place a virtual dragon on your real-world table without it floating away.

The Problem: The "First-Person" Blur

Imagine you are wearing a camera on your head, walking through a busy kitchen. You grab a mug, walk past a table, and turn around quickly. To a computer, this is a nightmare. The view spins wildly, your hand blocks the view of the mug, and objects pop in and out of sight. Existing methods for building 3D maps often get confused here. Some assume everything is still (like a museum), while others only track things you are currently touching. They struggle to keep a consistent "memory" of every object in the room, especially when the camera is moving fast and the view is messy.

The Solution: EgoTrack3D

Enter EgoTrack3D, a new system designed to be the ultimate "memory keeper" for first-person videos. Think of it as a super-organized librarian for a chaotic library. Its job is to take a shaky, fast-moving video and turn it into a stable, 3D map of every object it sees, whether that object is a stationary chair or a cup being carried around.

The system works in two main modes, depending on how much information it has:

The "High-Definition" Mode (Dense): If the computer has perfect 3D depth data (like a super-accurate 3D scanner), it acts like a master sculptor. It takes 2D outlines of objects from the video and "lifts" them into 3D space, building a full point-cloud model of every item. It then uses a special motion detector to figure out which objects are moving and which are staying put. If it sees the same object twice, it smartly merges the data so it doesn't count the same chair as two different chairs.
The "Real-World" Mode (Sparse): In the real world, perfect 3D scanners are rare. Often, the computer only has a rough guess at where things are. In this mode, EgoTrack3D switches tactics. Instead of trying to build a perfect 3D model of every pixel, it uses a "best guess" 3D box for each object. To handle the messiness of moving hands and shaky cameras, it uses a clever trick: it watches for interactions. If it sees a hand grabbing a pot, it knows that pot is "dynamic" and needs special care. It creates a temporary "anchor" to keep track of that pot even if the 3D box looks a bit wobbly.

How It Works: The Detective's Toolkit

The system uses a few clever tools to solve the puzzle:

The Motion Score: It doesn't just guess if something is moving; it tracks tiny points on the object. If those points shift significantly, the system flags the object as "moving" and updates its position quickly. If the points stay still, it treats the object as a solid, static part of the room.
The Voxel Merge: Imagine the 3D world is made of tiny 3D pixels called "voxels." If the system sees two objects occupying the same space, it knows they are likely the same object. It merges them together, cleaning up the map and removing duplicates.
The Interaction Guide: In the "Sparse" mode, when the computer is unsure about a moving object, it looks at the hands. If a hand is holding an object, the system gives that object a "VIP pass," ensuring it doesn't get lost or confused with other items, even if the 3D data is noisy.

The Results: A Clearer Picture

The researchers tested EgoTrack3D on a dataset called Aria Digital Twin (ADT), which contains videos of people moving around with perfect 3D data available. They compared their system against other top methods.

The results were promising. When using perfect data, EgoTrack3D improved the accuracy of locating objects by 11% compared to the strongest existing baseline. In the "High-Definition" mode, it achieved a score where 63.01% of the objects were correctly tracked and located (measured by a metric called PCL), which was significantly better than other methods that struggled with duplicates or lost tracks.

Even in the "Real-World" mode, where the data was messy and incomplete, the system showed it could still build a useful 3D map. It successfully tracked manipulated objects (like a pot or a notepad being moved) across large changes in viewpoint, a task where other systems often failed and lost track of the items entirely.

The Bottom Line

EgoTrack3D isn't a magic wand that solves every problem instantly. The authors admit that if the camera's depth data is very noisy, the system can still get confused, and it sometimes struggles to re-identify an object if it disappears for a long time. However, the paper suggests that by being modular—meaning it can swap out different tools depending on the quality of the input—it offers a robust step forward. It proves that we can build dynamic, 3D maps of our world from first-person videos, handling both the still furniture and the moving chaos of daily life, paving the way for smarter robots and more immersive augmented reality.

Technical Summary: EgoTrack3D

Problem Statement

Understanding 3D scenes from egocentric video is critical for robotics and autonomous navigation, yet it presents unique challenges due to rapid viewpoint changes, partial occlusions, truncated fields of view, and the frequent entry/exit of objects. While 2D scene graphs have advanced high-level reasoning, existing 3D tracking and scene graph construction methods often fail in dynamic egocentric settings. Current approaches typically assume static scenes, rely on precomputed representations, or focus only on predefined interacted objects. Consequently, there is a lack of methods capable of maintaining persistent, full-scene 3D tracking for both static and dynamic objects directly from egocentric RGB video without prior environmental models.

Methodology

The authors introduce EgoTrack3D, a modular framework designed to reconstruct and maintain a dynamic 3D scene representation directly from egocentric RGB video. The framework lifts 2D segmentation masks into a global 3D coordinate frame and tracks objects using geometric and visual features. The pipeline is divided into two variants to address different input constraints:

1. EgoTrack3D-Dense (Controlled Setting)

This variant assumes the availability of accurate per-pixel metric depth, camera poses, and segmentation masks (e.g., using ground truth from the Aria Digital Twin dataset).

Preprocessing: Generates segmentation masks and depth maps for each frame.
Motion Detection: Utilizes CoTracker3 to track an $N \times N$ grid of points over a time window ( $\Delta t$ ). It computes motion and direction scores for each point. Objects are classified as moving if their aggregated motion and direction scores exceed specific thresholds.
Appearance Embedding: Extracts per-object visual embeddings using MASA to support cross-frame association.
3D Lifting: Projects 2D image points to world coordinates using depth, intrinsics, and poses. For each object, 3D points are gathered, voxel-downsampled, and fitted with an oriented bounding box using Principal Component Analysis (PCA).
Tracking & Merging:
- Association: Matches new detections to existing tracks using a cost function combining 3D Intersection over Union (IoU), Chamfer distance (shape similarity), and feature similarity. The Hungarian algorithm solves the assignment.
- State Update: Static objects accumulate point clouds over time; moving objects update their point clouds dynamically. Visual features are exponentially smoothed.
- Track Merging: A voxel-based heuristic merges tracks with significant 3D point cloud overlap to eliminate duplicates caused by occlusions or partial views.

2. EgoTrack3D-Sparse (Real-World Setting)

This variant addresses the challenge of unreliable dense depth in unconstrained videos by replacing the dense lifting module with sparse 3D box estimation.

Input: Uses model-generated 2D masks (via CropFormer) and sparse point clouds.
3D Lifting: Employs BoxerNet to predict oriented 3D bounding boxes from sparse geometry.
Dynamic Association: Since BoxerNet predictions for hand-held objects can be noisy, the system uses interaction-guided dynamic association:
- Hands23 detects hands and their interacted objects.
- Masks overlapping with hand-interacted objects are marked as dynamic.
- A short-term SAM2 track is initialized for these dynamic masks and propagated for $K$ frames.
- Detections overlapping the propagated SAM2 mask are hard-associated to the same identity before geometric assignment, creating a robust 2D anchor for manipulated objects.
Merging: Duplicate tracks are merged based on 3D box and appearance similarity, except when protected by distinct dynamic-mask identities.

Key Contributions

Modular Pipeline: A framework for lifting 2D segmentations into temporally consistent 3D object tracks, adaptable to both dense and sparse 3D inputs.
Motion-Aware Association: A mechanism combining geometry (3D IoU, Chamfer distance) and appearance (visual features) to distinguish between static and dynamic objects, enabling selective updates.
Sparse-Input Variant: A robust configuration using BoxerNet-based 3D boxes and interaction-guided dynamic association, allowing the system to function without dense metric depth.
Comprehensive Evaluation: Evaluation on the Aria Digital Twin (ADT) and HD-EPIC datasets, demonstrating improved 3D tracking consistency under both accurate and noisy input conditions.

Results

Dense Input (ADT): EgoTrack3D-Dense achieved an 11% improvement in Percentage of Correct Locations (PCL) relative to the strongest baseline, Boxer, on the ADT dataset. Specifically, it achieved an average PCL of 63.01% compared to Boxer's 56.53% (note: Boxer is static-focused). OSNOM, another baseline, scored significantly lower at 21.41%. The track-merging heuristic was shown to be critical, improving the average PCL of the system by 83.38% relative to the variant without merging (34.36% vs. 63.01%).
Sparse Input (ADT): When replacing dense depth with sparse inputs, EgoTrack3D-Sparse improved the F1 score from 29.88% (Boxer baseline) to 40.79%. While aggregate metrics were dominated by static objects, the interaction-guided components proved essential for maintaining the identity of manipulated objects.
Qualitative (HD-EPIC): On real-world unconstrained videos, the sparse variant successfully preserved the identities of manipulated objects (e.g., a notepad and a pot) across large viewpoint changes, whereas removing interaction-guided association led to lost or incorrectly associated tracks.

Significance and Claims

The paper claims that EgoTrack3D addresses the gap in persistent full-scene tracking for dynamic egocentric observations. Unlike prior works that focus on static localization or restricted sets of interacted objects, EgoTrack3D constructs a 3D representation of the entire environment directly from RGB video without requiring prior environment models.

The authors emphasize the framework's modularity: it can operate with high-fidelity ground-truth inputs to isolate tracking performance or adapt to realistic, noisy sparse inputs by leveraging interaction cues. While the system does not generate a scene graph directly, it provides the necessary 3D structural representation (objects, attributes, spatial relationships) that can be extended into a scene graph. The work represents a step toward generalizable dynamic 3D scene reconstruction for robotics, augmented reality, and embodied spatial reasoning.

Limitations Acknowledged:
The authors note that the system inherits errors from perception modules (masks, depth, pose). In the dense setting, results reflect performance under accurate geometric inputs rather than fully end-to-end real-world perception. In the sparse setting, the system remains sensitive to errors in BoxerNet predictions for hand-held objects. Furthermore, association relies heavily on geometric costs, making re-identification difficult when 3D locations shift significantly without sufficient visual evidence.