💻 computer science

Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction

This paper proposes a three-step hierarchical Transformer that explicitly separates temporal encoding, multimodal fusion, and social interaction reasoning to achieve state-of-the-art pedestrian trajectory prediction performance on real-world datasets while improving scalability and interpretability.

Original authors: Raphaël Delécluse, Hazem Wannous, Laurent Grisoni, Laurent Guimas

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raphaël Delécluse, Hazem Wannous, Laurent Grisoni, Laurent Guimas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a busy train station, trying to guess where a specific person will be in the next few seconds. To do this well, you need to look at three different things:

How they are moving right now (Are they walking fast? Stopping?).
What their body language says (Are they turning their head? Are they holding a suitcase that suggests they might stop?).
What everyone else around them is doing (Is a crowd blocking their path? Is a friend waving them over?).

Most computer programs that try to predict this get overwhelmed. They try to look at all three things at once, which makes the math messy, slow, and hard to understand.

This paper introduces a new computer model called the Three-Step Hierarchical Transformer. Think of it as a team of three specialized detectives working in a line, rather than one detective trying to do everything at once.

The Three-Step Detective Team

Step 1: The "Motion Tracker" (Temporal Encoder)
First, the model looks only at the person's path so far. It ignores their body language and ignores other people. It's like a runner who just watches the track.

What it does: It asks, "Based on where this person has been for the last few seconds, where are they likely to go next?"
The Analogy: Imagine a weather forecaster who only looks at the wind speed. They can predict if a storm is coming, but they don't know if it's raining yet. This step creates a "rough draft" of the future path.

Step 2: The "Body Language Reader" (Modality Decoder)
Next, the model takes that rough draft and refines it using extra clues. It looks at the person's pose (are they turning their head?), their bounding box (how big are they?), or other visual details.

What it does: It asks, "The motion tracker said they will go straight, but look! They are turning their head left. Let's adjust the path."
The Analogy: This is like a detective who sees a suspect clutching a map. Even if the suspect was walking straight, the map suggests they might stop at a corner. The model uses a "smart summary" (a lightweight GRU) to quickly digest these body clues without getting bogged down in too much data.

Step 3: The "Crowd Manager" (Scene Transformer)
Finally, the model looks at the whole scene. It takes the refined path of the target person and compares it with the paths of everyone else in the crowd.

What it does: It asks, "If this person goes left, will they bump into someone? Is that group of people moving together?" It adjusts the path to make sure the person doesn't walk through a wall or a friend.
The Analogy: This is like a conductor in an orchestra. The conductor doesn't just listen to one violin; they listen to how the violin fits with the drums and the flutes. If the drums get loud, the violin might need to play softer. Here, the model ensures the person's path makes sense in the context of the whole crowd.

Why This Design is Special

1. It's Efficient (Saves Energy)
If you tried to do all three steps at once (looking at time, body language, and the whole crowd simultaneously), the computer would have to do a massive amount of math, like trying to solve a giant puzzle where every piece connects to every other piece. This gets too slow and expensive.

The Paper's Claim: By breaking it into three steps, the model only does the heavy math when it's necessary. It's like sorting mail: first by country, then by city, then by street. It's much faster than trying to sort every letter by every possible address at once.

2. It's Flexible (Handles Missing Info)
Sometimes a camera might miss a person's face, or the video might be blurry.

The Paper's Claim: Because the steps are separate, if the "Body Language" step misses a clue, the model can still use the "Motion Tracker" and "Crowd Manager" steps to make a good guess. It doesn't crash just because one piece of information is missing.

3. It's Clear (Easy to Understand)
Because the steps are separate, researchers can look at the model and say, "Ah, the error happened in Step 2, not Step 3." This makes it easier to fix and improve.

The Results: Did it Work?

The authors tested this "Three-Step Team" on three different datasets (simulated video games and real-world videos of people walking in cities and indoors).

The Outcome: The model performed better than almost all other existing methods. It was particularly good at predicting where people would end up (Final Displacement Error) and how far off the prediction was on average (Average Displacement Error).
Real-World Proof: It worked well in crowded, messy real-world environments (like the JRDB and Urban datasets), not just in perfect, clean simulations.
Specific Win: The paper notes that the model is especially good at spotting "early turning" behaviors—predicting that someone is about to turn before they actually start turning, thanks to the body language clues.

Summary

In short, this paper proposes a smarter way for computers to guess where pedestrians will go. Instead of a chaotic "kitchen sink" approach that throws everything into one big mix, it uses a three-step assembly line:

Watch the movement.
Read the body language.
Check the crowd.

This approach is faster, uses less computer power, and produces more accurate predictions for how people move in our busy world.

Technical Summary: Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction

Problem Formulation

Pedestrian trajectory prediction is a critical task for autonomous navigation, crowd analytics, and safety monitoring. The challenge lies in simultaneously modeling three complex factors: temporal dynamics (motion evolution), multimodal cues (e.g., body pose, bounding boxes, velocity), and social interactions (inter-agent influences) within crowded, cluttered environments.

Existing approaches often address these factors in isolation or entangle them within single, computationally expensive attention blocks. Trajectory-only models struggle with intent cues revealed by body articulation, while multimodal models often rely on simplistic fusion strategies. Furthermore, merging temporal, multimodal, and social signals into a unified attention mechanism frequently leads to poor scalability, reduced interpretability, and high computational costs due to quadratic complexity relative to the number of agents, time steps, and modalities.

Methodology

The authors propose a Three-Step Hierarchical Transformer that explicitly decouples reasoning into three complementary levels: temporal encoding, multimodal fusion, and scene-level interaction reasoning. This structured factorization aims to improve scalability, flexibility, and interpretability.

1. Temporal Encoder (Step 1)

Function: Processes each pedestrian's trajectory independently to capture motion dynamics (velocity trends, directional changes) without considering other modalities or agents.
Mechanism: Uses a Transformer operating along the time dimension. It takes observed trajectory tokens and appends learnable future query tokens to extrapolate an initial future hypothesis over the prediction horizon ( $T + \Delta$ ).
Input: Trajectory features projected into a latent space with composite positional encoding (sinusoidal for time, learned for agent identity).
Output: An initial trajectory hypothesis for each pedestrian.

2. Modality Decoder (Step 2)

Function: Refines the initial trajectory hypotheses by conditioning them on auxiliary modalities (e.g., 3D pose, bounding boxes).
Mechanism: Employs a decoder-style cross-attention mechanism.
- Queries: The temporally encoded trajectory embeddings.
- Keys/Values: Modality embeddings. To ensure efficiency, each modality sequence is first summarized into a single token per pedestrian using a lightweight GRU (Gated Recurrent Unit). This avoids attention over full temporal sequences for every modality.
Robustness: Handles missing or variable modalities naturally through attention masking.
Output: Refined trajectory embeddings enriched with semantic and kinematic cues.

3. Scene Transformer (Step 3)

Function: Models inter-pedestrian interactions at the scene level to adjust individual predictions based on social context.
Mechanism: Operates on a flattened set of time–agent tokens ( $N \times (T + \Delta)$ ). A Transformer encoder performs self-attention across all pedestrians and time steps.
Logic: Since the tokens already encode future motion hypotheses, the attention mechanism allows the model to reason about forthcoming social events (collisions, following, group formation) and adjust trajectories accordingly.
Output: Socially refined embeddings passed to a lightweight MLP prediction head for the final trajectory.

Computational Efficiency

The architecture addresses the quadratic complexity of standard Transformers ( $O(L^2)$ where $L = T \times N \times M$ ) by factorizing attention:

Temporal: $O(N \cdot L_1^2)$ where $L_1 = T + \Delta$ .
Cross-modal: $O(N \cdot L_1 \cdot M)$ (linear in modalities due to GRU summarization).
Scene: $O(L_3^2)$ where $L_3 = N \cdot (T + \Delta)$ .
This design ensures the model scales efficiently with the number of modalities and agents while maintaining expressiveness.

Key Contributions

Novel Architecture: A three-step hierarchical Transformer that decouples temporal, multimodal, and social reasoning, offering a modular and interpretable framework.
Efficient Fusion Strategy: Introduction of a computationally efficient multimodal fusion method using GRU summarization and decoder-style cross-attention, enabling scalable integration of heterogeneous cues.
Comprehensive Evaluation: Extensive quantitative, qualitative, and ablation studies demonstrating state-of-the-art (SOTA) performance on real-world datasets and competitive results on synthetic data.

Experimental Results

The model was evaluated on three datasets: JTA (synthetic), JRDB (real-world indoor/outdoor), and Pedestrians and Cyclists in Road Traffic (Urban).

Quantitative Performance:
- JRDB: Achieved SOTA results, reducing Average Displacement Error (ADE) from 0.37m (best baseline) to 0.33m and Final Displacement Error (FDE) from 0.72m to 0.67m.
- Urban: Showed substantial improvement over trajectory-only methods (0.51m ADE vs. ~0.57m) and multimodal baselines, reaching 0.96m FDE.
- JTA: Achieved competitive performance (0.96m ADE, 1.91m FDE) against top multimodal baselines.
Ablation Studies: Confirmed that the full hierarchical model outperforms variants lacking multimodal cues or social reasoning. Specifically, social reasoning significantly reduced endpoint errors, while multimodal cues improved temporal anticipation.
Qualitative Analysis: The model demonstrated the ability to anticipate complex behaviors like early turning (leveraging pose cues) and interaction-driven adjustments (yielding, following). However, the authors note that in scenarios with multiple feasible continuations (e.g., at bends), predictions may follow plausible but non-ground-truth paths, reflecting inherent uncertainty.

Significance and Claims

The paper claims that structured, hierarchical modeling provides a powerful foundation for trajectory prediction. By separating the reasoning processes, the model achieves:

Interpretability: Clear distinction between how time, modality, and social context influence predictions.
Scalability: Efficient handling of variable numbers of pedestrians and modalities without the prohibitive cost of joint attention.
Robustness: The ability to operate reliably even when specific modalities are missing or noisy.

The authors conclude that their approach effectively balances computational efficiency with the need to model complex, multi-faceted human motion, validating the benefits of decomposing the prediction task into specialized reasoning stages. Future work is suggested to extend this framework to probabilistic prediction and richer scene-aware modeling.