Technical Summary: LAMP - Localization Aware Multi-camera People Tracking in Metric 3D World

Problem Statement

Tracking 3D human motion from egocentric multi-camera headsets presents unique challenges that render existing monocular or static-camera methods ineffective. The authors identify three primary obstacles:

1. Severe Egomotion: Headsets undergo rapid, constant 6-DoF motion from the wearer's head, breaking algorithms that assume static or slow-moving cameras.
2. Complex Multi-View Dynamics: Modern headsets utilize multiple cameras with partial stereo overlaps. Observations frequently switch between cameras, and individuals may be partially or fully occluded by single or multiple cameras over time. Existing methods, often designed for monocular input, struggle with these "camera hand-off" scenarios and suffer from scale ambiguity.
3. Data Scarcity: Training data annotated with 3D human motion for specific multi-camera headset configurations is sparse and expensive to collect. Synthetic data often lacks realistic camera motion, and hardware configurations change frequently across generations, making it impractical to build device-specific datasets.

Methodology

The paper proposes LAMP (Localization Aware Multi-camera People Tracking), a framework that tracks multiple people directly in a metric 3D world by disentangling observer motion from target motion. The method follows a "lift-then-fit" paradigm:

1. Early World-Space Ray Lifting

Instead of processing raw pixels or 2D keypoints directly, LAMP leverages the known 6-DoF camera poses and calibration (available via state-of-the-art VIO or SLAM systems on modern headsets) to perform an early transformation:

• 2D Detection: 2D bounding boxes and keypoints are detected independently per camera and per person.
• Ray Lifting: These 2D keypoints are back-projected into 3D rays using the known camera intrinsics and extrinsics.
• World Alignment: The rays are transformed into a unified, gravity-aligned world reference frame. This step factors out the headset's egomotion, allowing the subsequent network to focus solely on learning human motion priors.
• Spatio-Temporal Association: 2D detections are associated across time and cameras using a bipartite matching algorithm (Hungarian algorithm) based on projected 3D points, creating consistent tracklets even across camera switches.

2. LAMP-Net (Spatio-Temporal Transformer)

The core of the system is LAMP-Net, an end-to-end trained spatio-temporal transformer.

• Input: A sequence of 3D ray clouds (Plücker rays) representing the observed keypoints over a temporal window (e.g., 4 seconds).
• Architecture: The network uses a transformer encoder-decoder structure. The encoder performs self-attention across spatial (joints) and temporal (frames) dimensions. The decoder utilizes cross-attention at each encoder block to iteratively aggregate motion and geometry information.
• Output: The network regresses parameterized 3D human body motion (SMPL parameters: pose, shape, global rotation, and translation) for each timestamp.
• Training Strategy: LAMP-Net is trained on simulated data. By lifting 2D keypoints to 3D rays, the model can be trained using arbitrary existing 3D motion datasets (e.g., Nymeria, AMASS) projected onto virtual camera layouts. This allows the model to generalize to unseen headset configurations (e.g., training on Gen1 data and testing on Gen2 hardware) without requiring raw video data for the target device.

3. Inference and Smoothing

The system employs a sliding-window inference strategy. To improve stability and reduce jitter, predictions for a specific timestamp are averaged over the overlapping windows of the sliding process. This provides a tunable trade-off between latency and accuracy.

Key Contributions

1. World-Space Ray Lifting Paradigm: The authors introduce a novel formulation that lifts 2D observations to 3D rays in a world frame before spatio-temporal reasoning. This explicitly disentangles camera motion from human motion, enabling robust tracking in dynamic egocentric settings.
2. Simulation-Ready Training: By operating on 3D rays rather than raw pixels, the method allows for the simulation of training data for arbitrary multi-camera configurations. This addresses the data scarcity problem, enabling zero-shot generalization to new headset layouts.
3. Multi-Camera Tracking in Metric 3D: LAMP is the first method to persistently track multiple people in a metric 3D world using multi-camera headsets, effectively handling camera hand-offs and partial occlusions.
4. State-of-the-Art Performance: The method achieves superior results in world-space localization (Root Trajectory Error, Jitter) compared to existing baselines, while matching or exceeding monocular baselines in local pose accuracy.

Results

The authors evaluate LAMP on the EMDB and Nymeria datasets, as well as real-world data from Project Aria Gen2 headsets.

• Quantitative Performance:
  • On EMDB (slow-moving cameras), LAMP significantly outperforms baselines (WHAM, GVHMR, PromptHMR) in Root Trajectory Error (RTE) and Jitter, demonstrating superior world-space stability. While it trails slightly in local pose metrics (MPJPE) on this specific dataset, the authors attribute this to the design choice of collapsing images to rays for multi-view aggregation.
  • On Nymeria (dynamic egocentric motion), LAMP outperforms PromptHMR across all metrics in both monocular and multi-camera configurations. The multi-camera setup (LAMP-mv) yields the best results, with a W-MPJPE (World-MPJPE without alignment) of 113.3mm compared to 203.4mm for the monocular variant.
• Coverage: Increasing the number of cameras significantly improves tracking coverage. In a social interaction scenario, coverage rises from 47% (1 camera) to 81% (4 cameras).
• Generalization: The model trained on simulated Gen1 data successfully tracks people on Aria Gen2 hardware (which has a different camera configuration), validating the effectiveness of the ray-based simulation approach.
• Real-Time Performance: The system runs in real-time on a single RTX 4090 GPU, capable of tracking multiple people with low latency.

Significance and Claims

The paper claims that LAMP provides a simple yet flexible solution to the long-standing challenge of tracking people in dynamic, egocentric environments. By accepting known 6-DoF poses as input and factoring them out early, the method avoids the complexity of jointly estimating camera and human motion.

The authors emphasize that their approach:

• Solves the Scale Ambiguity: By operating in a metric world frame, LAMP resolves the scale issues inherent in monocular 3D tracking.
• Enables Cross-Device Usage: The ability to simulate training data for arbitrary rig layouts means the system can be adapted to new headset generations without collecting massive new datasets.
• Facilitates Social Understanding: By providing persistent, metric 3D tracking of multiple people, LAMP enables higher-level social understanding for augmented reality assistants, a critical component for future context-aware AI systems.

The work is presented as a foundational step toward robust, real-time, multi-person tracking in the wild, specifically tailored for the capabilities of modern egocentric devices.