HiMAP: History-aware Map-occupancy Prediction with Fallback

HiMAP is a tracking-free trajectory prediction framework that leverages history-aware occupancy maps and a query-based retrieval mechanism to deliver robust, multi-modal motion forecasts for autonomous driving, effectively serving as a reliable fallback when traditional multi-object tracking fails.

The Gist

Imagine you are driving a self-driving car. To drive safely, the car needs to predict where other cars, pedestrians, and cyclists will go in the next few seconds.

Most current self-driving systems work like a strict librarian. They try to assign a unique name tag (an ID) to every person or car they see. Once a car is named "Car #42," the system follows "Car #42" continuously, remembering its entire history to guess where it will go next.

The Problem:
In the real world, things get messy. A car might drive behind a truck (occlusion), a sensor might glitch, or two cars might merge so closely that the system loses track of who is who. When the "librarian" loses the name tag, the whole prediction system panics. It forgets the car's history and starts guessing wildly, which is dangerous.

The Solution: HiMAP
The paper introduces HiMAP, a new system that doesn't need name tags at all. Think of HiMAP not as a librarian, but as a detective looking at a security camera feed.

Here is how HiMAP works, using simple analogies:

1. The "Ghost Map" (Historical Occupancy)

Instead of trying to keep a list of "Car #42," HiMAP looks at the road like a heat map or a "ghost trail."

• The Analogy: Imagine walking through a foggy forest. You can't see the person ahead clearly, but you can see the footprints they left in the mud and the bent grass where they passed.
• How it works: HiMAP takes every single "snapshot" of the road and paints a picture of where something was. It doesn't care if it was Car #42 or Car #99; it just knows, "At this spot, at this time, there was a vehicle." It builds a timeline of these "ghost footprints" on the map.

2. The "Time-Traveling Detective" (Historical Query)

When the system needs to predict where a specific car is going, it doesn't ask, "Who is this?" Instead, it asks, "Where does the pattern match?"

• The Analogy: Imagine you see a person walking away from you. You don't know their name, but you see their current stride and direction. You look at the "ghost footprints" on the ground and say, "Ah, the footprints from 5 seconds ago match the size and direction of this person's current step. So, that must be where they came from."
• How it works: HiMAP looks at the car's current state (speed, direction) and scans the "Ghost Map" of the past. It iteratively digs through the history to find the specific trail that belongs to this car, effectively reconstructing its past path without ever needing a name tag.

3. The "Crystal Ball" (Future Prediction)

Once HiMAP has reconstructed the car's history using the ghost footprints, it feeds that story into a prediction engine.

• The Analogy: Now that the detective knows exactly how the person was walking, how fast they were moving, and where they turned previously, they can make a very educated guess about where the person will step next.
• The Result: Even if the "librarian" (the tracking system) has completely failed and lost the car, HiMAP can still say, "I know where this car came from, so I know exactly where it's going."

Why is this a big deal?

• The Safety Net: If the main tracking system breaks (like a GPS losing signal), HiMAP acts as a backup parachute. It ensures the car doesn't freeze or make a dangerous guess just because it lost a name tag.
• The Performance: The researchers tested this on a massive dataset (Argoverse 2). When tracking failed, standard systems crashed. HiMAP, however, kept performing almost as well as the best systems that do have perfect tracking.
• The Speed: It works instantly. It doesn't need to wait for the tracking system to "recover" and re-identify the car. It provides stable predictions the moment a car is detected.

In Summary

HiMAP is like a self-driving car that has amnesia about names but perfect memory of patterns. It doesn't need to know "This is Bob's car." It just needs to know, "Something was here, then there, then there," and based on that pattern, it can safely predict the future. This makes self-driving cars much safer in the messy, unpredictable real world where sensors fail and cars get lost.

Technical Summary

1. Problem Statement

Autonomous driving systems rely heavily on Multi-Object Tracking (MOT) to provide identity-consistent historical trajectories for motion forecasting. However, tracking is inherently brittle in real-world scenarios due to:

• Occlusions and dense traffic.
• Identity switches (ID switches) and drift.
• Missed detections or flickering.

When tracking fails, standard predictors (which assume continuous, labeled tracks) suffer a catastrophic drop in performance, leading to unsafe decision-making. Existing methods often rely on affinity scores or intermediate tracker outputs to bridge gaps, but few address the scenario where tracking is completely unavailable or unstable. The core challenge is to predict future trajectories accurately using only unlabeled, per-frame detections without relying on persistent agent IDs.

2. Methodology: HiMAP Framework

HiMAP is a tracking-free trajectory prediction framework designed to serve as a robust fallback. It reconstructs agent history implicitly from raw detections using a spatiotemporally invariant representation. The architecture consists of four main modules (see Fig. 2 in the paper):

A. Spatiotemporally Invariant Scene Encoding

Unlike agent-centric methods that re-normalize the scene to the target agent's coordinates at every timestep (preventing context reuse), HiMAP adopts the QCNet approach. It encodes agents and HD map elements (lanes) in a global, invariant coordinate frame. This allows historical data to be stored and reused efficiently without requiring ID-based trajectory storage.

B. Historical Occupancy Map Encoder

Since tracking IDs are unavailable, the model converts past detections into occupancy maps:

• For each historical timestep, agent embeddings are linked to nearby lane segments via a graph structure.
• A gating mechanism (using a Sigmoid-activated MLP) modulates the influence of agents on lane occupancy based on their relative positions. This filters out irrelevant agents and focuses on those interacting with the lane dynamics.
• The output is a sequence of occupancy tensors $[T, M, D]$ representing the scene's history without explicit agent identities.

C. Historical Query Module

This is the core innovation for implicit tracking. It reconstructs the specific agent's history from the unlabeled occupancy maps:

1. Temporal Map Embedding: A GRU summarizes the sequence of historical occupancy maps into a compact temporal embedding.
2. Query Initialization: A query vector is initialized by cross-attending the current agent state with the temporal map embedding.
3. Iterative Reconstruction: The query is iteratively updated by attending to historical occupancy maps in reverse temporal order.
   • At each step, the query retrieves agent-specific information from the unlabeled map representation.
   • An MLP projects the updated query into a local displacement vector.
   • This process effectively "unrolls" an RNN through time, reconstructing a coherent historical trajectory for the target agent without ever knowing its ID.

D. Future Trajectory Decoder

The decoder uses a DETR-style architecture:

• It takes the reconstructed historical trajectory, the final historical query, the current agent state, and the map context.
• $K$ learnable mode queries decode $K$ diverse future trajectory hypotheses with associated probabilities.
• The model uses an anchor-free design, which the authors found sufficient and more efficient than the two-stage anchor-based approach used in some baselines.

3. Key Contributions

1. Tracking-Free Robustness: Proposes the first framework explicitly designed to operate reliably when tracking fails, reconstructing history solely from unlabeled detections.
2. Historical Query Mechanism: Introduces a novel module that iteratively queries historical occupancy maps conditioned on the current state to implicitly recover agent-specific trajectories.
3. Spatiotemporal Invariance: Leverages invariant encodings to enable streaming inference and efficient reuse of map context, avoiding the computational overhead of re-normalization.
4. Practical Fallback: Demonstrates that a tracking-free system can match the performance of tracking-based methods in ideal conditions while significantly outperforming them when tracking breaks.

4. Experimental Results

The method was evaluated on the Argoverse 2 and Argoverse 1 datasets.

• Performance under No-Tracking:

  • When tracking IDs are removed, standard baselines (e.g., QCNet) degrade significantly (e.g., minFDE6 jumps from 1.29 to 3.23).
  • HiMAP achieves minFDE6 = 1.33 and minADE6 = 0.68 in the no-tracking setting.
  • Compared to a fine-tuned QCNet baseline (trained without tracking), HiMAP yields relative improvements of 11% in FDE, 12% in ADE, and a 4% reduction in Miss Rate (MR).
  • HiMAP's performance is comparable to state-of-the-art tracking-based methods that assume perfect tracking.

• Safety Implications (The "Recovery Gap"):

  • Analysis shows that when tracking fails, standard trackers require ~13–14 steps (1.3–1.4 seconds) of recovery time to regain HiMAP's level of accuracy.
  • During this gap, vehicles travel significant distances (up to 30m for fast agents), posing severe safety risks. HiMAP provides immediate, stable predictions without this recovery lag.

• Ablation Studies:

  • Removing the Historical Query Module causes performance to collapse.
  • Using occupancy maps is superior to direct state-to-state querying (which suffers from exponential complexity and noise).
  • 30 reconstructed historical timesteps was found to be the optimal balance between context and noise.

5. Significance

HiMAP addresses a critical gap in autonomous driving safety: the assumption of perfect perception. By decoupling trajectory prediction from the reliability of multi-object tracking, HiMAP acts as a complementary safety mechanism.

• Reliability: It ensures that the prediction module does not fail catastrophically when the perception stack (tracking) is compromised.
• Efficiency: It supports streaming inference via reusable encodings.
• Real-world Applicability: The results suggest that for safety-critical autonomy, systems should not rely solely on tracking-based forecasts but must incorporate robust, tracking-agnostic fallbacks like HiMAP to handle inevitable perception failures.

The code is publicly available at: https://github.com/XuYiMing83/HiMAP.