💻 computer science

Recall to Predict: Grounding Motion Forecasting in Interpretable Motion Bank

The paper proposes "Recall to Predict," an end-to-end differentiable framework that enhances motion forecasting interpretability and accuracy by grounding predictions in a contrastively learned "motion bank" of physically realizable trajectories, which are dynamically retrieved via a novel Anchor Retrieval Layer and refined by a specialized decoder to eliminate opaque latent queries while achieving competitive multi-modal performance on Argoverse 2 and Waymo Open Motion datasets.

Original authors: Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius Zöllner

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhishek Vivekanandan, Ahmed Abouelazm, J. Marius Zöllner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. The hardest part isn't just knowing how to steer; it's predicting what other cars and pedestrians will do next. Will that car turn left? Will that pedestrian step off the curb?

Most current AI models try to guess these future paths by starting with a "blank slate." They try to invent a path from scratch every time. The problem is, this often leads to a "black box" situation where the AI makes a guess, but no one knows why it made that guess, or if the guess is even physically possible (like a car driving through a wall).

The paper "Recall to Predict" (R2P) proposes a smarter way to do this. Instead of inventing paths from nothing, the AI recalls them from a library.

Here is the breakdown of how it works, using simple analogies:

1. The "Motion Bank" (The Library of Moves)

Imagine a massive library, but instead of books, it contains millions of real, recorded driving paths (trajectories). These are real examples of cars turning, stopping, or merging, all checked to make sure they are physically possible.

The Old Way: The AI tries to draw a new path on a blank piece of paper every time.
The R2P Way: The AI looks at the current situation and says, "Hey, this looks like that situation I saw in the library yesterday. Let me pull that specific path out of the library and use it as a starting point."

2. The "Anchor Retrieval Layer" (The Librarian)

This is the brain of the operation. When the AI sees a new scene (a busy intersection, for example), it doesn't just guess. It sends out a "query" to the Motion Bank.

The Analogy: Think of this like a very smart librarian. You tell the librarian, "I'm at a red light, and there's a car to my left." The librarian doesn't just hand you any book; they find the exact few books (or "anchors") that match your specific situation.
The Magic: The paper introduces a special trick (called a "Straight-Through Gumbel-Softmax") that allows the computer to pick a specific book from the shelf while still being able to "learn" from the choice. Usually, picking a specific item stops the learning process, but this trick keeps the learning flowing smoothly.

3. The "Dual-Level Gating" (The Smart Filter)

Once the librarian finds potential paths, the system needs to decide which ones are actually useful.

The Analogy: Imagine you are listening to a conversation in a noisy room. You have to decide: "Do I listen to my friend? Do I listen to the traffic noise? Or do I ignore the background chatter?"
The system uses a "gating" mechanism to weigh different pieces of information. It might decide, "In this situation, the car next to me is the most important thing to pay attention to, so I'll focus 40% of my attention there," while ignoring irrelevant traffic lights or distant cars. This ensures the AI focuses on what actually matters.

4. The "Refinement Decoder" (The Fine-Tuner)

The paths pulled from the library are good, but they aren't perfect. They are like a rough sketch.

The Analogy: Imagine you found a great recipe in a cookbook (the library), but you need to adjust the salt because you are using a different brand of tomatoes.
The AI takes the "library path" and makes tiny, precise adjustments (offsets) to fit the exact current situation. Crucially, the paper says the AI is forced to make these adjustments small. It can't use the adjustments to fix a bad library choice; it must pick a good library path first, then just tweak it. This keeps the AI honest and interpretable.

Why is this a big deal?

No More Black Boxes: Because the AI picks a real, human-observed path from the library first, we can look at the library and say, "Ah, the AI chose the 'turn left' path because it saw a car waiting to turn." We can see why it made the decision.
Safety: Since the paths come from a library of real, physically possible moves, the AI is less likely to suggest impossible or dangerous maneuvers.
Diversity: It prevents the AI from getting stuck in a rut (where it always predicts the same thing). By pulling from a diverse library, it can predict many different possibilities (e.g., the car might turn, or it might stop).

The Results

The authors tested this on two major driving datasets (Argoverse 2 and Waymo Open Motion). They found that their system:

Was just as accurate (or better) at predicting where cars would go compared to the state-of-the-art models.
Did this while using much less map data (only 1/4th of what other models use).
Provided a clear, transparent view of how it made its decisions, rather than hiding them inside a complex math "black box."

In short, Recall to Predict teaches the self-driving AI to stop guessing from thin air and start learning from a curated, organized library of real-world driving experiences, making it both smarter and easier to understand.

Technical Summary: Recall to Predict (R2P)

Problem Statement

Motion forecasting in autonomous driving faces a fundamental trade-off between predictive accuracy and interpretability. Standard anchor-based architectures often rely on opaque latent queries that are prone to "latent collapse" (where diverse queries converge to similar representations) or utilize naive trajectory sampling that fails to capture rare, safety-critical edge cases. While existing methods improve multi-modal diversity, they frequently sacrifice the ability to inspect the model's reasoning process during intermediate prediction steps. Furthermore, many approaches rely heavily on rigid heuristics or abstract endpoint generation, which can propagate perception noise and lack grounding in physically realizable behaviors.

Methodology

The authors propose Recall to Predict (R2P), an end-to-end differentiable framework that grounds trajectory forecasting in a pre-trained, structured "motion bank." Rather than regressing paths from a blank slate, the model dynamically retrieves explicit, physically realizable motion priors.

1. Pre-trained Motion Bank

The core of the framework is a library of prototypical movements constructed via contrastive learning (based on embeddings from prior work [26]). This bank maps geometrically similar trajectories to proximal points in a latent space, creating a structured feature space where each trajectory (anchor) is linked to a specific embedding.

2. Anchor Retrieval Layer

A novel Anchor Retrieval Layer acts as an auxiliary encoder to select candidate anchors from the motion bank:

Context-Aware Query Adaptation: Orthogonally initialized latent queries ( $Q_{base}$ ) are adapted using a Dual-Level Gated Cross-Attention mechanism. This processes heterogeneous inputs (agent history, neighbor states, map polylines) through independent pathways. A micro-level sigmoid gate determines how much context each query absorbs, while a macro-level softmax (including a learnable "null" sink) balances relevance across modalities.
Differentiable Selection: To bridge the gap between historical context and future trajectory geometries, the system computes cosine similarity between adapted queries and frozen bank embeddings. A Straight-Through Gumbel-Softmax (STE) estimator is employed to perform discrete trajectory selection. This allows the network to select exact, unblended anchors in the forward pass while maintaining continuous gradient flow for backpropagation.

3. Factorized Scene Encoder & Decoder

Encoder: The architecture uses a factorized-then-fused strategy. It independently encodes dynamic agents (social interactions) and static map elements (lane topology and traffic lights) before fusing them. This produces a focal agent token representing the entire scene context.
Decoder: A DETR-style decoder iteratively refines the retrieved anchor tokens. Crucially, the queries are initialized with the retrieved anchors, ensuring a strict 1-to-1 correspondence between latent queries and physical trajectories. The decoder performs cross-attention first with the target's dynamic history and then with the environmental context.
Prediction Heads: Parallel MLP heads predict continuous kinematic states, mode confidence, and spatial offsets. The offset head specifically predicts residuals from the pre-decoder anchor tokens, preventing the decoder from lazily compensating for poor anchor selections.

4. Learning Objectives

The model is trained with a composite loss function:

Winner-Takes-All (WTA) Kinematic Loss: Minimizes error for the best-performing mode using a Negative Log-Likelihood (NLL) of a bivariate Gaussian, augmented with velocity and heading penalties.
Dual-Objective Endpoint Loss: Applies soft-min weighting to the endpoints of refined anchors, guiding both discrete selection and local geometric accuracy.
Latent Diversity Loss: Enforces orthogonality among the adapted queries to prevent retrieval collapse, ensuring selected anchors represent distinct semantic intents (e.g., left turn vs. straight).

Key Contributions

End-to-End Framework with Motion Bank: Introduction of a framework that injects explicit, physically realizable motion priors into the decoding process, enabling the retrieval of context-relevant driving behaviors.
Differentiable Anchor Retrieval Layer: A novel module that decouples forward discrete selection from backward continuous gradients via STE, providing highly interpretable intermediate representations that expose the model's decision-making process.
Multi-Objective Refinement Strategy: A combination of WTA kinematic GMM, offset regularization, and latent diversity constraints that geometrically fine-tunes selected anchors while strictly preserving their foundational semantic intent.

Experimental Results

The framework was evaluated on the Argoverse 2 (AV2) and Waymo Open Motion (WOMD) datasets.

Argoverse 2: R2P-M achieved a minFDE6 of 1.33, outperforming established baselines like HPTR (1.43) and ForecastMAE (1.42). It also achieved a competitive brier-minFDE of 2.03.
Waymo Open Motion: R2P-M achieved a minADE6 of 0.5474 and minFDE6 of 1.1132, surpassing strong baselines including EDA and MTR++.
Efficiency: The model operates with significantly reduced context sizes (256 map polylines vs. 1024 in HPTR) and eliminates the need for Non-Maximum Suppression (NMS) or complex post-processing heuristics.
Ablation Findings: Experiments confirmed that structured, cluster-based sampling of the motion bank is critical for stable representation learning, whereas random sampling led to degraded metrics and failure to converge on appropriate priors.

Significance and Claims

The paper claims that R2P successfully resolves the historical trade-off between predictive accuracy and multi-modal interpretability. By replacing opaque latent queries with a contrastively-trained motion bank, the framework grounds trajectory generation in a structured space of physically realizable behaviors.

The authors emphasize that the Anchor Retrieval Layer eliminates the "black box" nature of standard forecasting architectures. By exposing the exact motion primitives the model considers before the decoding phase, the approach provides transparency into the network's intermediate reasoning. The paper asserts that state-of-the-art multi-modal forecasting can be achieved without sacrificing human-interpretable reasoning, offering a paradigm shift for autonomous driving where safety-critical decisions can be inspected and understood.

Note: The authors acknowledge a current limitation regarding Vulnerable Road Users (VRUs), noting that the current motion bank is biased toward vehicular kinematics, resulting in coarser representations for pedestrians and cyclists.