🤖 machine learning

Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling

The paper introduces U2Diffine, a unified diffusion model that simultaneously performs multi-agent trajectory completion and forecasting while providing state-wise heteroscedastic uncertainty estimates and error probability rankings for generated modes, outperforming state-of-the-art methods across four sports datasets.

Original authors: Guillem Capellera, Antonio Rubio, Luis Ferraz, Antonio Agudo

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Guillem Capellera, Antonio Rubio, Luis Ferraz, Antonio Agudo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a chaotic basketball game. Players are sprinting, passing, and dodging each other. Now, imagine you are a coach trying to predict where everyone will be in the next few seconds. Or, imagine you are a video editor trying to fix a glitchy recording where the camera briefly lost track of a player, and you need to "fill in the missing frames" so the video looks smooth again.

This paper introduces a new computer brain (an AI model) called U2Diffine (and its faster cousin, U2Diff) designed to do exactly that: predict and repair the movement paths of multiple people (or objects) at the same time.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: Guessing the Future (and the Past)

Most AI models that predict movement are like fortune tellers who only look at the past to guess the future. They say, "Based on where you were, you will likely go here."

The Flaw: They often ignore the "what ifs." In a real game, a player might stop, turn, or get blocked. The AI usually just gives one "best guess" path, or a few random guesses, without telling you how confident it is in those guesses.
The Missing Piece: If the AI is unsure, it should say, "I'm not sure, the path could be anywhere in this big circle." If it's sure, it should say, "I'm very sure, the path is this tiny line." This paper calls that "uncertainty."

2. The Solution: The "Uncertainty-Aware" Diffusion Model

The authors built a system based on Diffusion Models. Think of diffusion like a sculptor working with clay.

The Process: Imagine you have a perfect statue (the real movement path). The AI first turns it into a blob of noise (random static). Then, it learns how to slowly turn that noise back into the statue, step-by-step.
The Twist: Usually, the sculptor just tries to make the statue look right. The authors taught their sculptor to also keep a mental note of how shaky their hand is at every step.
- If the hand is shaky, the AI draws a big, fuzzy cloud around the path.
- If the hand is steady, it draws a tight, precise line.
The Result: The AI doesn't just give you a path; it gives you a path plus a "confidence map" showing exactly where the prediction is risky and where it is safe.

3. Two Versions: The "Precision" vs. The "Speedster"

The authors created two versions of this AI to suit different needs:

U2Diffine (The Precision Sculptor): This version is very careful. It uses complex math (called a "first-order Taylor approximation") to calculate exactly how the uncertainty spreads from the noise back to the real world. It's like a master sculptor who measures every millimeter. It is the most accurate but takes longer to run.
U2Diff (The Speedster): This version skips the complex math to save time. It makes a smart guess about the uncertainty without doing all the heavy calculations. It's about 4 times faster than the precision version and is almost as good at predicting the path, though slightly less precise on the "confidence map."

4. The "Ranking" Assistant (RankNN)

Sometimes, the AI generates 20 different possible futures (20 different "what if" scenarios). How do you know which one is the most likely to happen?

The Old Way: You might just pick the one that looks the "smoothest."
The New Way (RankNN): The authors added a special "judge" (a neural network) that looks at all 20 scenarios and assigns each one a score of "how likely it is to be wrong."
The Analogy: Imagine a sports analyst watching 20 different game replays. Instead of just guessing, this analyst looks at the players' movements and the "shakiness" of the prediction to say, "Scenario #3 has a 90% chance of being right, while Scenario #15 is probably wrong." This helps pick the best prediction automatically.

5. Where Did They Test It?

They didn't test this on medical data or self-driving cars (unless the paper says so, which it doesn't). They tested it strictly on sports data:

Basketball: Tracking 10 players and a ball.
Football (American): Tracking 22 players and a ball.
Soccer: Tracking 22 players and a ball.

6. The Big Win

The paper claims that their method is better than the current best methods (State-of-the-Art) in two main ways:

Accuracy: It predicts where players will be more accurately, especially when trying to "fix" missing parts of a video (trajectory completion).
Reliability: It is much better at knowing when it is unsure. If the AI says "I'm not sure," it usually means the situation is actually chaotic or hard to predict. This makes the system much more trustworthy for real-world applications like fixing sports video footage.

In short: They built a smarter way for computers to watch sports, predict player movements, and fix broken video recordings, all while honestly admitting when they are guessing and when they are sure.

Technical Summary: Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling

Problem Statement
Multi-agent trajectory modeling has traditionally focused on forecasting future states, often neglecting broader tasks such as trajectory completion (inferring missing or unobserved portions of a trajectory). Existing methods, including those based on GANs, CVAEs, and standard Diffusion Probabilistic Models (DDPMs), typically predict agent states without providing state-wise measures of heteroscedastic uncertainty. Furthermore, popular multi-modal sampling methods lack error probability estimates for each generated scene under the same prior observations, making it difficult to rank prediction reliability at inference time. A critical gap remains in the ability to quantify the likelihood of each predicted state and to rank multiple generated modes based on their proximity to ground truth.

Methodology
The authors introduce U2Diffine, a unified diffusion model designed to perform trajectory completion while simultaneously offering state-wise heteroscedastic uncertainty estimates. The framework is built upon the following core components:

Unified Diffusion Framework: Inspired by CSDI (Conditional Score-based Diffusion Imputation), the model uses a bidirectional MambaSSM for temporal processing and a Social Transformer for agent interactions, replacing traditional Transformer encoders to better handle temporal dynamics without fixed positional encodings.
Heteroscedastic Uncertainty Estimation: Unlike standard denoising losses that predict a single noise vector, U2Diffine models the reverse noise as a bi-variate Gaussian distribution. It learns both the mean noise state ( $\epsilon^\mu_\theta$ ) and a $2 \times 2$ covariance matrix ( $\epsilon^\Sigma_\theta$ ) for each agent at every timestep. This is achieved by augmenting the standard denoising loss with a Negative Log-Likelihood (NLL) term, allowing the model to capture data-dependent (aleatoric) uncertainty, including directional dependencies and rotated uncertainty ellipses.
Reverse Gaussian Sampling (U2Diffine): To propagate latent space uncertainty to the real state space, the authors derive a first-order Taylor approximation. This method linearizes the score network around the current mean to compute the Jacobian matrices ( $J_s$ ), which are used to update the covariance matrix during the reverse sampling process. This provides precise, robust, and consistent state-wise uncertainty estimates without resorting to costly Monte Carlo sampling.
Accelerated Baseline (U2Diff): To address the computational cost of gradient computation required for the Jacobian, the authors propose U2Diff. This variant assumes a zero-gradient ( $J_s = 0$ ) and uses a simpler uni-variate uncertainty model. To mitigate potential variance inflation, it employs a "variance delay" strategy, initiating uncertainty propagation only after an intermediate denoising step. This approach reduces inference time by approximately 4x while maintaining competitive displacement error.
Rank Neural Network (RankNN): To address the challenge of ranking multiple generated modes, the authors introduce a post-processing supervised architecture. RankNN takes the generated modes, their predicted means, and variances as input. It processes these through a Social-temporal Block and a Multi-scene Transformer to output an error probability score for each mode. The model is trained to maximize the Spearman correlation between these predicted scores and the actual Scene Average Displacement Error (SADE).

Key Contributions

Unified Framework: A diffusion-based approach for multi-agent trajectory completion that integrates forecasting, imputation, and inferring unobserved agents within a single framework adaptable to variable numbers of agents and time steps.
State-wise Uncertainty: Augmentation of the diffusion framework with a bi-variate negative log-likelihood loss to enable heteroscedastic uncertainty estimation for each predicted latent state.
Analytical Uncertainty Propagation: Development of a first-order Taylor approximation for Reverse Gaussian Sampling, propagating latent uncertainty to the output space to yield precise state-wise estimates.
Error Probability Estimation: A novel post-processing supervised architecture (RankNN) that provides error probability estimates for each generated mode, demonstrating high correlation with ground truth errors.
Efficiency: The proposal of U2Diff, a faster variant that avoids gradient computation during sampling, offering $\sim4\times$ faster inference with comparable displacement error.

Experimental Results
The method was evaluated on four challenging sports datasets: NBA, Basketball-U, Football-U, and Soccer-U.

Trajectory Completion: U2Diffine and U2Diff outperformed state-of-the-art methods (including UniTraj) across all three completion datasets. Notably, U2Diffine achieved approximately 33% and 44% improvements in minSADE20 on the Football-U and Soccer-U datasets, respectively.
Trajectory Forecasting: On the NBA dataset, the unified approach achieved state-of-the-art performance in scene-level metrics (minSADE20/minSFDE20), outperforming leading methods like LED, MART, and MoFlow by more than 3%, despite using a straightforward i.i.d. sampling strategy.
Uncertainty Quality: The models achieved high Accuracy Rates (AccRate), with U2Diffine reaching over 95% on Football-U and Soccer-U datasets. The bi-variate modeling allowed for the prediction of rotated uncertainty ellipses that aligned well with spatial data distributions.
Ranking Performance: The RankNN framework significantly improved the correlation between predicted uncertainty and actual error. While simple uncertainty metrics (AvgUcty) showed moderate correlation (Spearman $\rho \approx 0.3$ ), the RankNN-predicted error probabilities achieved high correlations ( $\rho$ ranging from 0.58 to 0.79). This enabled effective ranking of generated modes, reducing Top-1 scene error significantly compared to random or variance-based selection.

Significance
The paper claims that this work establishes a new benchmark for trajectory completion and forecasting by addressing the critical lack of uncertainty quantification in multi-agent settings. By providing state-wise heteroscedastic uncertainty and a mechanism to rank generated modes by error probability, the approach enhances the reliability of generated trajectories in complex interactive environments like sports. The authors emphasize that while U2Diffine offers superior likelihood estimation through rigorous analytical propagation, U2Diff provides a practical, high-speed alternative for applications where inference latency is critical. The integration of RankNN further bridges the gap between generative diversity and practical utility by allowing users to select the most reliable predictions from a set of generated modes.