🤖 AI

TrajMamba: An Ego-Motion-Guided Mamba Model for Pedestrian Trajectory Prediction from an Egocentric Perspective

The paper proposes TrajMamba, an ego-motion-guided Mamba model that effectively predicts pedestrian trajectories from an egocentric perspective by explicitly modeling the complex relative motion between the ego-vehicle and pedestrians, achieving state-of-the-art performance on the PIE and JAAD datasets.

Original authors: Yusheng Peng, Gaofeng Zhang, Liping Zheng

Published 2026-03-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yusheng Peng, Gaofeng Zhang, Liping Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking down a busy street while holding a camera. You are trying to guess where a specific stranger is going to walk next. But here's the catch: you are also moving.

If you stand still, it's easy to guess where the stranger is going. But if you are walking forward, backward, or turning, the stranger's movement on your camera screen becomes a confusing mix of their walking and your walking. It's like trying to guess the path of a fish in a river while you are also drifting in a boat.

This is the exact problem the paper TrajMamba solves. It's a new computer program designed to predict where pedestrians will go, specifically from the perspective of a moving car or robot (the "ego" perspective).

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "Double-Motion" Blur

Most old systems look at traffic from a drone's view (looking straight down). It's like watching a chessboard from above; you can clearly see where every piece is moving.

But self-driving cars and robots see the world like a human does: from the front.

The Pedestrian is moving.
The Car (Camera) is moving.
The Result: The image on the screen is a superposition of both. If the car speeds up, the pedestrian looks like they are slowing down or moving backward, even if they are standing still.

Old methods tried to solve this by just "mixing" the two movements together in a big bowl of data. They said, "Here is the car's speed, and here is the person's speed; let's guess the future." The problem is, this mixing often gets messy, and the computer struggles to understand how the car's movement specifically changes the person's path.

2. The Solution: TrajMamba (The "Smart Navigator")

The authors built a new system called TrajMamba. Think of it as a highly trained navigator who doesn't just look at the data but understands the relationship between the driver and the pedestrian.

They use a new type of AI brain called Mamba.

The Analogy: Imagine LSTM and Transformer (the old AI brains) are like a student trying to read a long book by memorizing every single word. It works, but it's slow and gets tired.
Mamba is like a student who understands the flow of the story. It can read long sequences (like a long video) very quickly and remember the important parts without getting overwhelmed. It's faster and smarter at spotting patterns in time.

3. How TrajMamba Works: The Three-Step Dance

The system has three main parts, working together like a relay race:

Step 1: The Two Observers (Encoders)
The system has two "eyes" looking at the data separately:

Eye A (Pedestrian Encoder): Watches the person walking. It uses Mamba to figure out, "Is this person speeding up? Are they turning? What is their rhythm?"
Eye B (Ego-Motion Encoder): Watches the car. It uses Mamba to figure out, "Are we accelerating? Are we braking? Are we turning left?"
Why separate? Because you need to understand the two movements clearly before you can figure out how they interact.

Step 2: The Smart Guide (The Decoder)
This is the most important innovation. In old systems, the two "eyes" just dumped their notes into a pile.
In TrajMamba, the system works like a Director and an Actor:

The Pedestrian's history is the "Actor" (the context).
The Car's movement is the "Director" (the guide).
The Director says to the Actor: "Okay, you were walking straight, but because I (the car) just turned right, here is how your path on the screen will change."
Instead of mixing the data, the system explicitly asks: "How does the car's movement regulate or change the pedestrian's movement?" This allows the AI to predict the future much more accurately.

Step 3: The Crystal Ball (Generator)
Finally, the system takes this clear understanding of the relationship and draws the future path. It doesn't just guess a random spot; it calculates the exact box where the person will be in 0.5, 1.0, or 1.5 seconds.

4. The Secret Sauce: "Predicting the Difference"

The paper also uses a clever trick called Residual Prediction.

Old Way: Try to predict the exact future coordinates from scratch. (Like trying to guess the exact temperature tomorrow without knowing today's weather).
TrajMamba Way: It assumes the person will keep walking at their current speed (a "Constant Velocity" guess). Then, it only predicts the small difference (the error) between that simple guess and reality.
Analogy: Instead of trying to draw a perfect mountain from scratch, you draw a flat hill first, and then just add the "bumps" and "peaks" where they actually are. It's much easier and more accurate.

5. The Results: Why It Matters

The authors tested this on two famous datasets (PIE and JAAD), which are like huge libraries of real-world driving videos.

The Score: TrajMamba beat almost every other method. It made fewer mistakes in predicting where the pedestrian would be.
The Efficiency: Because it uses Mamba, it does this faster and with less computer power than the older, heavier models.

Summary

TrajMamba is like a super-smart co-pilot for self-driving cars. It doesn't just watch the pedestrian; it watches the pedestrian through the eyes of the moving car. By separating the two movements and then explicitly teaching the AI how the car's motion changes the pedestrian's path, it can predict the future with incredible accuracy, keeping our roads safer.

1. Problem Statement

The paper addresses the challenge of pedestrian trajectory prediction from an egocentric perspective (first-person view), which is critical for autonomous driving and robot navigation.

Core Challenge: Unlike third-person (bird's-eye view) predictions, egocentric prediction must account for the complex dynamic relative motion between the moving camera (ego-vehicle) and the tracked pedestrian. The apparent motion in the video frame is a superposition of the pedestrian's intrinsic movement and the camera's movement.
Limitations of Existing Methods: Current approaches typically use early or late fusion strategies to combine pedestrian motion and ego-motion features into a unified latent representation before decoding. This paradigm often results in ambiguous modeling of the relative motion relationship, making it difficult for the model to explicitly analyze how ego-motion dynamically regulates or modulates pedestrian movement.

2. Methodology: TrajMamba

The authors propose TrajMamba, a novel framework based on the Mamba (State Space Model) architecture. The model consists of four main modules:

A. Input Representation & Preprocessing

Differentiated Motion Representation: Instead of using raw bounding box coordinates, the model computes the center point displacement (velocity) and scale variation ( $\Delta w, \Delta h$ ) between adjacent frames.
CV-CS Assumption & Residual Prediction: The model adopts a Constant Velocity and Constant Scaling (CV-CS) assumption. It generates a reference trajectory based on the average velocity and scale variation of the last 5 frames. The network is then tasked with predicting the residual offset ( $\Delta$ ) between the ground truth and this reference trajectory. This injects physical priors into the learning process and eases optimization.

B. Architecture Components

Pedestrian Motion Encoder (PME):
- Uses an MLP to embed the pedestrian's motion state sequence.
- Employs a Mamba encoder to extract temporal features, leveraging its linear time-series modeling capability and efficiency in handling long sequences.
Ego-Motion Encoder (EME):
- Takes vehicle state information (speed or behavior labels) as input.
- Uses an MLP for embedding and a Mamba encoder to extract ego-motion features.
Ego-Motion-Guided Decoder (EMGD):
- Key Innovation: Unlike traditional fusion methods, this decoder treats pedestrian motion features as historical context and ego-motion features as explicit conditional cues for future prediction.
- The decoder uses a Mamba layer to sequentially model how the ego-motion dynamically modulates the pedestrian's movement, explicitly capturing the relative motion relationship rather than just fusing features.
Future Trajectory Generator (FTG):
- An MLP maps the decoded future features into the final predicted bounding box offsets, which are added to the reference trajectory to produce the final output.

3. Key Contributions

Novel Framework: Proposes TrajMamba, the first framework to utilize the Mamba architecture specifically for egocentric pedestrian trajectory prediction.
Explicit Modeling Mechanism: Introduces an Ego-Motion-Guided Decoder that explicitly models the relative motion between the pedestrian and the camera by using ego-motion as a guiding condition rather than a fused feature. This allows for a clearer disentanglement of motion sources.
Efficiency and Performance: Demonstrates that Mamba offers superior efficiency (lower parameters and FLOPs) compared to LSTM, GRU, and Transformer baselines while achieving state-of-the-art accuracy.
Residual Prediction Strategy: Develops a CV-CS based residual prediction method that improves physical plausibility and prediction smoothness.

4. Experimental Results

The model was evaluated on two standard datasets: PIE (Pedestrian in Traffic) and JAAD (Joint Attention for Autonomous Driving).

Metrics: Average Displacement Error (ADE), Final Displacement Error (FDE), Average Root Mean Square Error (ARB), and Final Root Mean Square Error (FRB).
Performance on PIE (1s prediction):
- TrajMamba achieved 7.72 ADE and 16.99 FDE, outperforming the previous best (PedFormer) by significant margins (e.g., ~40% improvement in ADE).
- It also outperformed general models like BiTrap and SGNet.
Performance on JAAD (1s prediction):
- Achieved 13.90 ADE and 30.76 FDE, surpassing PedFormer by ~22% in ADE.
Ablation Studies:
- Decoder Mechanism: The Ego-Motion-Guided Decoder (EMGD) consistently outperformed the classic Post-Fusion Decoding (PFD) mechanism across all backbone types (LSTM, GRU, Transformer, Mamba).
- Backbone Efficiency: Mamba-based models achieved the best accuracy with the lowest computational cost (e.g., 6.35G FLOPs and 0.20M parameters for the full model on PIE), significantly outperforming Transformer and RNN variants.
- Input/Output: The proposed CV-CS residual representation yielded the lowest errors compared to raw coordinate or velocity-only inputs.

5. Significance

Paradigm Shift: Moves away from implicit feature fusion toward explicit conditional modeling of ego-motion, providing a more interpretable and accurate way to handle the dual-motion problem in egocentric views.
Efficiency: Proves that State Space Models (Mamba) are highly effective for sequential motion prediction, offering a lightweight alternative to heavy Transformer architectures without sacrificing accuracy.
Application: The method directly enhances the safety and navigation capabilities of autonomous vehicles and mobile robots operating in dynamic, crowded environments where the sensor platform itself is moving.

Future Work: The authors note that current work does not account for interactions between multiple pedestrians in crowds and plan to incorporate pedestrian-pedestrian interaction modeling in future iterations.