🤖 AI

RetroMotion: Retrocausal Motion Forecasting Models are Instructable

RetroMotion is a transformer-based motion forecasting model that decomposes complex multi-agent predictions into marginal and joint distributions using a retrocausal information flow, achieving state-of-the-art performance on major datasets while uniquely supporting scene-adaptive instruction following.

Original authors: Royden Wagner, Omer Sahin Tas, Felix Hauser, Marlon Steiner, Dominik Strutz, Abhishek Vivekanandan, Jaime Villa, Yinzhe Shen, Carlos Fernandez, Christoph Stiller

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Royden Wagner, Omer Sahin Tas, Felix Hauser, Marlon Steiner, Dominik Strutz, Abhishek Vivekanandan, Jaime Villa, Yinzhe Shen, Carlos Fernandez, Christoph Stiller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing at a busy intersection, watching cars, pedestrians, and cyclists move around. Predicting where everyone will go in the next few seconds is incredibly hard. If you try to guess the exact path of every single person at once, the number of possibilities explodes, like trying to solve a puzzle where every piece can be a million different shapes.

The paper "RetroMotion" introduces a new way for computers to solve this puzzle. Here is the breakdown of their approach using simple analogies:

1. The Two-Step Dance: Solo vs. Group

Instead of trying to predict the entire chaotic crowd all at once, the authors split the problem into two parts:

The Solo Act (Marginal Forecasts): First, the computer looks at each person individually. It asks, "If this car were alone on the road, where would it go?" It creates a "solo plan" for everyone.
The Group Dance (Joint Forecasts): Next, the computer looks at the people who are interacting (like a car waiting for a pedestrian to cross). It takes those "solo plans" and remixes them to see how they fit together.

The Magic Trick (Retrocausality):
Usually, you predict the future based on the past. But this model uses a clever trick called "retrocausality." Think of it like writing a story. Usually, you write the beginning, then the middle, then the end. But this model writes the ending first (based on the solo plans), and then uses that ending to rewrite the beginning of the group story.

Why do this? It's like realizing, "Oh, if that car ends up turning left, it must have started slowing down earlier." By knowing the destination, the model can better understand the start of the interaction. This makes the initial solo guesses less stressful to get perfect, because the "group dance" step can fix any small mistakes.

2. The "Fuzzy" Map (Uncertainty)

When predicting where a car will be, you can't just draw a single thin line. You need to show a "cloud" of possibilities because the driver might swerve or brake.

The Old Way: Most models assume this cloud is a perfect circle (Normal distribution) or a specific diamond shape (Laplace distribution).
The RetroMotion Way: The authors say, "Let's not force the cloud into a specific shape." Instead, they use a flexible shape called an Exponential Power Distribution.
The Analogy: Imagine trying to fit a cloud of smoke into a glass jar. Some clouds are round, some are flat, some are spiky. Instead of forcing the smoke into a round jar, this model molds the jar to fit the smoke perfectly. They found that these "smoke clouds" usually look a bit like diamonds (Laplace) but with a tiny bit of roundness mixed in, which makes them much more accurate.

3. The Compression Trick (DCT)

Predicting a path for 8 seconds involves hundreds of data points. Storing all those points is heavy and slow.

The Analogy: Imagine you have a long, wiggly rope. Instead of storing the exact position of every inch of the rope, you describe it using a few simple waves (like a musical chord).
The model uses a mathematical tool called Discrete Cosine Transform (DCT) to compress the path into a few "waves." This makes the computer run faster and ignores tiny, noisy jitters that don't matter, focusing only on the smooth, real movement.

4. The "Instructable" Feature (The Surprise)

The most surprising part of the paper is that the model can take instructions, even though the researchers didn't explicitly teach it to do so.

The Experiment: The researchers took the model's prediction for a car and manually changed the end of the path to say, "Go to this specific spot" (a goal-based instruction).
The Result: The model didn't just blindly follow the new path. It looked at the new instruction and said, "Okay, if the car needs to go there, it must have started turning earlier," and it adjusted the whole path to make sense with the traffic rules and other cars.
The "Turn Left" Test: They even tried to force a car to "Turn Left" into oncoming traffic (which is illegal). The model looked at the instruction, realized it was dangerous, and said, "No, I can't do that," and adjusted the car's path to stay in its own lane instead.
The Takeaway: By simply training the model to be good at predicting traffic, it accidentally learned how to listen to commands and adapt them to the real world.

Summary

RetroMotion is a traffic prediction system that:

Breaks big problems into small solo plans, then mixes them into a group plan.
Uses a "backwards-looking" trick (using the end to fix the start) to make the group plan smarter.
Uses flexible "clouds" to predict uncertainty, rather than rigid shapes.
Compresses data to run fast.
Surprisingly, learned to listen to human instructions and adapt them to the scene, all without being explicitly taught how to follow orders.

The authors tested this on real-world driving data (Waymo, Argoverse, V2X) and found it predicts traffic better than previous methods and handles complex interactions very well.

1. Problem Statement

Motion forecasting for autonomous driving involves predicting the future trajectories of multiple road users (agents). This task faces two primary challenges:

Exponential Complexity: The joint distribution of trajectories grows exponentially with the number of agents. Modeling the full joint distribution for all agents in a scene is computationally prohibitive.
Interaction Modeling: While marginal distributions (predicting each agent independently) are efficient, they fail to capture complex interactions between agents. Conversely, existing joint modeling methods often suffer from high inference latency or are limited to small numbers of agents.
Lack of Control: Standard motion forecasting models are typically "black boxes" that cannot be easily guided by high-level instructions (e.g., "turn left" or "yield to pedestrians") without retraining.

2. Methodology

The authors propose RetroMotion, a transformer-based framework that decomposes multi-agent motion forecasting into marginal and joint components, linked by a unique retrocausal information flow.

A. Decomposition Strategy

The model separates the task into two stages:

Marginal Forecasting: A decoder predicts independent trajectory distributions for all agents in the scene.
Joint Forecasting: A second decoder generates joint distributions specifically for interacting agents.
- Retrocausal Flow: Instead of predicting joint trajectories directly from raw inputs, the model re-encodes the marginal distributions to generate joint distributions. This creates a flow of information from the "future" (marginal predictions) back to the "earlier" joint modeling stage.
- Pairwise Modeling: The joint decoder transforms trajectories into a local reference frame and uses attention mechanisms to model interactions. It approximates the full joint distribution ( $K^2$ combinations) by focusing on diagonal query pairs, compressing information into $K$ pairs while allowing off-diagonal pairs to update diagonal ones via attention.

B. Probabilistic Modeling

Compressed Exponential Power Distributions: The model does not assume standard Gaussian or Laplace distributions. Instead, it models positional uncertainty using a mixture of bivariate Normal and Laplace distributions (approximating an Exponential Power distribution).
DCT Compression: To reduce computational complexity and remove high-frequency noise, the location parameters (means $\mu$ ) of the distributions are compressed using Discrete Cosine Transforms (DCT). The model predicts a small set of DCT coefficients, which are then reconstructed via Inverse DCT (IDCT) to form the full trajectory.

C. Architecture

Scene Encoder: Processes multimodal inputs (past trajectories, lane data, traffic lights) using polylines and transformer encoders to generate global scene context embeddings.
Two Decoders:
- Marginal Decoder: Predicts per-agent trajectory densities.
- Joint Decoder: Takes the re-encoded marginal features and scene context to predict joint interaction distributions.
Loss Function: Trained via Maximum Likelihood Estimation (MLE) using a multitask loss that combines negative log-likelihoods for both marginal and joint forecasts, optimized only on the trajectories closest to the ground truth.

3. Key Contributions

Retrocausal Information Flow: The method connects marginal and joint forecasts by re-encoding marginal predictions. This reduces the burden on initial marginal forecasts and, crucially, creates an interface for issuing instructions. By modifying marginal trajectories before the joint decoding stage, the model can adapt its joint predictions to follow specific goals.
Compressed Exponential Power Distributions: The authors demonstrate that modeling uncertainty with learned exponential power distributions (approximated as Normal-Laplace mixtures) outperforms standard Normal or Laplace assumptions.
Emergent Instruction Following: The paper shows that standard motion forecasting training implicitly enables the model to follow goal-based and directional instructions without explicit instruction-tuning. The model adapts basic commands (e.g., "turn left") to the scene context (e.g., avoiding oncoming traffic).
Efficiency and Scalability: The method supports forecasting for up to 8 agents with low inference latency (~60ms on A6000 GPUs), significantly faster than diffusion-based joint modeling methods.

4. Experimental Results

The model was evaluated on Waymo Open Motion, Argoverse 2, and V2X-Seq datasets.

Waymo Interaction Prediction Challenge:
- RetroMotion achieved a mAP of 0.2519 (SMoE hybrid configuration), outperforming strong baselines like MTR++ (0.2326) and QCNeXt (0.2352).
- It demonstrated that joint modeling significantly improves upon treating marginal predictions as joint predictions (mAP dropped from 0.2441 to 0.1797 when joint decoding was removed).
Cross-Dataset Generalization:
- Trained solely on Waymo data, RetroMotion generalized effectively to Argoverse 2 and V2X-Seq, outperforming specialized models like Wayformer and RedMotion in terms of minFDE and Miss Rate.
Instruction Following:
- Goal-Based: Modifying marginal trajectories to match desired goals reduced the final displacement error (minFDE) by 11.8%.
- Directional: When given a "turn left" instruction that violated traffic rules (e.g., turning into oncoming lanes), the model successfully adapted the joint trajectory to stay on the road, reducing overlap rates with other agents by 22% and increasing on-road probability by 33%.
Ablation Studies:
- Using Exponential Power distributions with DCT compression yielded the best results (mAP 0.195), outperforming Normal (0.172) and Laplace (0.176) distributions without compression.
- Analysis of mixture weights revealed the learned distributions are predominantly Laplace-like but benefit from a small Normal component.

5. Significance

RetroMotion represents a significant step toward interpretable and interactive autonomous driving systems.

Efficiency: It solves the scalability issue of joint forecasting by decomposing the problem and using retrocausal re-encoding.
Human-AI Alignment: The discovery that standard training yields instruction-following capabilities suggests that motion forecasting models can naturally learn to align with human intent. This opens the door for operators to guide vehicle behavior in edge cases or complex scenarios without retraining the model.
Robustness: The use of compressed probabilistic representations makes the model robust to dataset-specific noise and computationally efficient for real-time deployment.