EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Observations

This paper introduces EgoTraj-Bench, the first real-world benchmark for ego-centric trajectory prediction under noisy observations, and proposes BiFlow, a dual-stream flow matching model with an EgoAnchor mechanism that achieves state-of-the-art robustness by concurrently denoising historical inputs and forecasting future motion.

The Gist

Here is an explanation of the paper EgoTraj-Bench, broken down into simple concepts with everyday analogies.

The Big Picture: The "Blindfolded Navigator" Problem

Imagine you are trying to guide a robot through a busy coffee shop.

• The Old Way (Bird's-Eye View): Most robot scientists train their robots using a perfect, high-definition security camera looking straight down from the ceiling. From this view, the robot can see everyone clearly, knows exactly where they are, and never loses track of who is who. It's like playing a video game with a "God Mode" map.
• The Real World (Ego-View): In reality, robots don't have ceiling cameras. They have cameras on their own "heads" (like a GoPro). This view is messy. People walk behind pillars (occlusion), the camera shakes, the robot might get confused about which person is which (ID switch), and the edges of the camera lens distort the image.

The Problem: The robots are trained on the perfect "God Mode" data, but when they go into the real world with their shaky, messy "head-mounted" cameras, they get confused and crash. They can't predict where people will go because their input data is full of noise.

The Solution Part 1: EgoTraj-Bench (The New Training Ground)

The authors realized they needed a new way to train robots that mimics real life. They created EgoTraj-Bench.

• The Analogy: Imagine a driving school. Previously, they only let students practice on a perfect, empty track with perfect weather. Now, they built a simulator that puts the student in a car with a cracked windshield, foggy windows, and a GPS that sometimes glitches.
• How they did it: They took a dataset where they had both a perfect ceiling view (the "truth") and a messy robot-eye view (the "noise") of the same scene. They took the messy robot view, turned it into a map, and used it to train models, while using the perfect ceiling view to grade how well the model did.
• The Result: They proved that when you feed the "perfect" training data into the "messy" robot view, the robots fail miserably. This benchmark forces researchers to build robots that can handle real-world chaos.

The Solution Part 2: BiFlow (The "Double-Brain" Robot)

To fix the problem, the authors built a new AI model called BiFlow.

• The Analogy: Think of a detective trying to solve a crime.

  • Old Detectors: They look at the blurry, torn-up witness sketch and try to guess the future immediately. If the sketch is bad, the guess is bad.
  • BiFlow (The New Detective): This detective has a two-step process:
    1. Restoration: First, they look at the blurry, torn sketch and say, "Wait, let me clean this up first. I'll fill in the missing parts and fix the smudges."
    2. Prediction: Then, using the now-clean sketch, they predict where the suspect will go next.

• How it works: BiFlow runs two tasks at the same time. One part of its brain tries to "denoise" the past (fix the messy history), and the other part uses that cleaned-up history to predict the future. By fixing the past first, the prediction becomes much more accurate.

The Secret Sauce: EgoAnchor (The "Intuition" Module)

The model also includes a feature called EgoAnchor.

• The Analogy: Imagine you are walking in a crowd. Even if you can't see a person's face because they are behind a pole, you can guess where they are going based on their body language or the general flow of the crowd.
• How it works: EgoAnchor acts like a "gut feeling" or a compass. It looks at the messy history and extracts the "intent" (the general direction and goal) of the people. Even if the data is noisy, this "intent" helps the robot stay on track and not get thrown off by a single bad data point. It stabilizes the prediction, like a gyroscope on a ship.

The Results: Why This Matters

When they tested their new model (BiFlow) against the old ones using their new messy benchmark:

• The Old Models: Crashed and burned. Their predictions were way off because they couldn't handle the "noise."
• BiFlow: Performed significantly better (about 10–15% more accurate). It successfully "cleaned" the messy input and predicted the future path with high confidence.

Summary

This paper is about admitting that the real world is messy. Instead of pretending robots have perfect vision, the authors built a new test (EgoTraj-Bench) that forces robots to deal with bad data. They then built a new robot brain (BiFlow) that first cleans up the bad data and then uses "intuition" (EgoAnchor) to predict the future, making robots much safer and more reliable in crowded, real-world environments.

Technical Summary

Here is a detailed technical summary of the paper "EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Observations".

1. Problem Statement

Pedestrian trajectory prediction is critical for autonomous systems (e.g., mobile robots, service vehicles) to navigate safely in human-centric environments. However, a significant gap exists between current research and real-world deployment:

• Idealized Assumptions: Most existing methods are trained and evaluated using Bird's-Eye View (BEV) data, which assumes globally consistent, noiseless observations and flawless agent tracking.
• Real-World Reality: Autonomous agents typically rely on First-Person View (FPV/ego-centric) cameras. These observations suffer from inherent perceptual artifacts, including:
  • Occlusions: Pedestrians hidden behind others or objects.
  • ID Switches: Tracking algorithms swapping identities between agents.
  • FOV Truncation: Agents entering or exiting the camera's field of view.
  • Perspective Distortion: Geometric errors near image corners.
• The Consequence: Models trained on clean BEV data fail catastrophically when deployed with noisy FPV inputs, as they lack robustness to these structured perception errors. Existing synthetic benchmarks fail to capture the complexity of real-world noise.

2. Proposed Solution: EgoTraj-Bench and BiFlow

The paper introduces two core components: a new benchmark to evaluate robustness and a novel model architecture to address the noise.

A. EgoTraj-Bench (The Benchmark)

This is the first real-world benchmark specifically designed for trajectory prediction under ego-centric noisy observations.

• Data Source: Built upon the TBD dataset, which provides synchronized BEV (overhead) and FPV (robot-mounted) videos.
• Construction Pipeline:
  1. Noisy Input Generation: Pedestrian trajectories are extracted from raw FPV videos using detectors (YOLOv8) and trackers (BotSort). These trajectories inherently contain real-world noise (occlusions, ID switches, drift).
  2. Metric Projection: These noisy FPV trajectories are projected into the global BEV coordinate system using calibrated camera intrinsics and robot odometry.
  3. Clean Supervision: The noisy history is paired with clean, human-verified future trajectories extracted from the synchronized BEV overhead view.
• Significance: It allows for the fair evaluation of BEV-based models under realistic, noisy historical inputs, bridging the gap between simulation and deployment.

B. BiFlow (The Methodology)

To address the challenges identified by the benchmark, the authors propose BiFlow, a dual-stream flow matching model.

• Core Philosophy: Instead of treating history as a fixed, noisy input, BiFlow jointly learns to denoise the historical observations and predict the future trajectories. It leverages the latent relationship between reconstructing clean history and forecasting the future.
• Architecture Components:
  1. Contextual Encoder: A Transformer-based module that models social interactions and noisy dynamics using Multi-Head Self-Attention (MHSA) and agent identity embeddings. It processes the noisy history and a validity mask.
  2. EgoAnchor Mechanism: A novel intent-distillation module. It extracts compact "intent tokens" from the encoded history (both agent-level and scene-level) via attention mechanisms. These tokens are injected into the future decoder via feature-wise affine modulation (similar to FiLM), providing a robust prior to stabilize predictions even when input is partial or corrupted.
  3. Dual-Stream Flow Matching:
     • Stream 1 (Reconstruction): Reconstructs the clean historical trajectory ($X$) from the noisy input.
     • Stream 2 (Prediction): Predicts the clean future trajectory ($Y$).
     • Both streams share the encoder's latent representation but use separate decoders. The model is trained using a flow matching objective (conditioned on noise vectors) to generate diverse, multi-modal future trajectories.
• Inference: During testing, only the noisy history is input. The model uses the learned denoising capabilities and the EgoAnchor priors to directly predict the future, without explicitly outputting the reconstructed history.

3. Key Contributions

1. EgoTraj-Bench: The first real-world benchmark that aligns noisy, first-person visual histories with clean, metric-accurate future ground truth, enabling rigorous evaluation of robustness.
2. BiFlow Framework: A novel dual-stream flow matching architecture that jointly recovers noisy history and predicts the future, implicitly leveraging clean historical semantics to guide forecasting.
3. EgoAnchor: A mechanism to distill intent priors from historical features, stabilizing predictions under partial or corrupted inputs via feature modulation.
4. Empirical Insights: Demonstrated that state-of-the-art BEV models suffer significant performance degradation (up to 10–15% error increase) under ego-view noise, highlighting the necessity of noise-aware modeling.

4. Experimental Results

The authors evaluated BiFlow against various baselines (Recurrent models like VRNN, Transformer-based like TUTR, and Flow-based like MoFlow) on both the new EgoTraj-TBD and the synthetic T2FPV-ETH datasets.

• Performance Gains:
  • BiFlow achieved State-of-the-Art (SOTA) performance.
  • On average, it reduced minADE (Average Displacement Error) by 10–15% and minFDE (Final Displacement Error) by 13–15% compared to existing baselines.
  • Specifically on T2FPV-ETH, BiFlow achieved a minADE of 0.60m and minFDE of 0.74m, outperforming the previous best by >11% and >15% respectively.
• Robustness: The model showed superior stability when generating fewer trajectory candidates (lower $K$), indicating a more accurate predicted distribution.
• Ablation Studies: Removing the EgoAnchor or the Shared Encoder resulted in significant performance drops, confirming that joint learning of history denoising and intent distillation is crucial for robustness.
• Qualitative Analysis: Visualizations showed BiFlow successfully predicting physically plausible trajectories even when the input history was heavily occluded or contained ID switches, whereas baselines often produced erratic or collision-prone paths.

5. Significance and Impact

• Bridging the Reality Gap: This work challenges the community to move beyond idealized BEV assumptions. By providing a benchmark that reflects real-world sensor limitations, it sets a new standard for evaluating trajectory prediction systems intended for physical robots.
• Methodological Shift: It demonstrates that jointly modeling history reconstruction and future prediction is a more effective strategy for handling noisy inputs than simple pre-processing or correction modules (like CoFE).
• Practical Application: The proposed BiFlow and EgoAnchor mechanisms offer a pathway for deploying safer, more reliable autonomous agents in crowded, dynamic, and unstructured human environments where sensor data is inevitably imperfect.

In conclusion, the paper establishes that robust trajectory prediction requires acknowledging and modeling the specific nature of ego-centric noise, rather than ignoring it, and provides both the data and the algorithmic framework to achieve this.