💻 computer science

Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Prediction

This paper proposes a context-guided diffusion framework enhanced by an energy-based joint refinement mechanism to generate multi-agent motion predictions that are both diverse and interaction-consistent, achieving state-of-the-art performance on both marginal and joint metrics across multiple benchmarks.

Original authors: Lei Chu, Yuhuan Zhao

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lei Chu, Yuhuan Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a busy train station, trying to guess where everyone will be in the next few seconds. You have to predict the path of a single person, but you also have to guess how they will move together with the crowd. If you guess wrong about one person, you might predict a collision that never happens, or you might miss a group of friends walking together.

This paper, titled "Diverse Yet Consistent," introduces a new computer program called CODA designed to solve this exact problem. It helps AI predict how groups of people (or agents) will move in the future, ensuring the predictions are both varied (covering many possibilities) and coordinated (making sense as a group).

Here is how CODA works, broken down into simple concepts:

1. The Problem: The "Average" Trap and the "Chaotic" Mess

Previous AI models had two main problems:

The "Average" Trap: Older models often tried to predict just one path for everyone. It was like telling a crowd, "Everyone will walk exactly halfway between the left and right doors." In reality, some people go left, some go right, and some stop. Averaging them out creates a blurry, useless prediction.
The "Chaotic" Mess: Newer models (using something called "Diffusion") are great at creating many different possibilities (diversity). However, they sometimes act like a room full of people who don't know each other. The AI might predict Person A going left and Person B going right, even though they are holding hands and need to stay together. The individual paths look okay, but the group behavior is impossible.

2. The Solution: CODA's Three-Step Dance

CODA fixes this by using a three-part system to guide the AI's imagination.

Step 1: The "Context Guide" (DCGC)

Imagine you are trying to guess where a friend will go. You don't just look at where they are standing; you look at who they are with, where they are looking, and what the map looks like.
CODA does this by gathering rich context. It looks at the history of the movement and the interactions between people. It creates a "guide" that tells the AI, "Hey, remember, this person is in a hurry, and that group is sticking together." This ensures the AI doesn't just guess randomly but uses the clues from the past.

Step 2: The "Adaptive Mixer" (ACIM)

Now, the AI starts generating predictions. Think of this like a chef mixing ingredients.

Old way: The chef might add the "context" (the guide) too early or too late, ruining the flavor.
CODA's way: It uses a special "Adaptive Mixer" that constantly blends the context guide into the prediction process as it happens. It's like a DJ who seamlessly mixes a beat (the movement) with a specific song request (the context) in real-time. This ensures the predictions are diverse (many different songs) but still follow the request (the context).

Step 3: The "Group Harmony Check" (JDR)

This is the paper's secret sauce. After the AI generates a bunch of possible futures, it runs a final check called Joint Distribution Refinement.

The Analogy: Imagine the AI has generated 100 different scenarios of a dance floor. In 90 of them, two dancers are about to crash into each other. In 10, they are dancing perfectly together.
The Fix: CODA uses an "Energy-Based" system. Think of "Energy" as a measure of how "awkward" or "impossible" a scenario is. A collision has high energy (bad). Dancing together has low energy (good).
The system acts like a strict choreographer. It looks at all the generated scenarios and says, "We like the diversity, but we need to lower the 'awkwardness' score." It tweaks the predictions so that while everyone still has their own unique path, they no longer walk through each other. It keeps the individual paths realistic but forces the group to be consistent.

3. The Results: Better Than the Rest

The authors tested CODA on four different datasets, ranging from people walking in cities (ETH/UCY) to basketball players running on a court (NBA).

Accuracy: CODA was better at predicting exactly where a single person would end up (Marginal metrics) compared to almost all other top methods.
Consistency: It was also excellent at predicting how the group would move together (Joint metrics).
The Balance: The paper claims CODA is the first to really master the balance. It doesn't sacrifice the "group harmony" just to get the "individual path" right, and vice versa.

Summary

In short, CODA is a smart prediction engine that:

Listens to the history and social context (who is with whom).
Imagines many different possible futures (diversity).
Polishes those futures to ensure they make sense as a group (consistency), removing impossible collisions or weird behaviors.

The result is a system that can predict human movement in crowded, chaotic environments with high accuracy and realistic group behavior, outperforming previous methods on standard tests.

Technical Summary: CODA – Context-Guided Diffusion with Energy-Based Joint Refinement

Problem Statement

Human trajectory prediction in multi-agent environments is a critical task for applications such as autonomous driving, digital health, and human-robot interaction. The core challenge lies in generating predictions that are simultaneously diverse (capturing the multimodal nature of human behavior where one past can lead to multiple plausible futures) and jointly consistent (ensuring that the predicted trajectories of interacting agents do not collide or violate social norms).

Existing approaches face two primary limitations:

Generative Limitations: Classical models (e.g., VAEs, GANs) often struggle to capture multimodal distributions, leading to averaged trajectories and accumulated errors. While diffusion models naturally model multimodality, they can suffer from temporal inconsistencies (jitters) and goal ambiguity, producing locally plausible but globally inconsistent trajectories.
Evaluation Limitations: Most existing methods are evaluated using single-agent (marginal) metrics (ADE/FDE), which fail to fully reflect the joint dynamics of multi-agent interactions. Conversely, methods optimized for joint consistency often sacrifice marginal accuracy.

Methodology: CODA Framework

The authors propose CODA (Context-Guided Diffusion with Energy-Based Joint Refinement), a diffusion-based framework designed to balance trajectory diversity with multi-agent consistency. The framework consists of three key modules:

1. Dynamic Context as Guidance Condition (DCGC)

To enhance the expressiveness of generated motions, CODA extracts rich contextual information from historical trajectories to serve as guidance conditions.

Agent-wise Context: Uses self-attention on embedded historical trajectories to capture individual motion patterns and long-term intentions.
Global Context: Employs a Transformer-based architecture to extract dynamic interaction features (short-term interaction dynamics) among agents.
Integration: These features are concatenated to form a comprehensive guidance condition $G$ that captures both individual intent and mutual interactions.

2. Adaptive Condition Integration Module (ACIM)

Standard diffusion methods often use MLPs or Transformers that may introduce bias or limit diversity when integrating conditions. CODA introduces ACIM to dynamically inject the guidance condition $G$ into the noisy target embedding via cross-attention.

This mechanism strengthens conditional guidance during the diffusion process by fusing the guidance-enhanced representation with the original noisy embedding, allowing the model to fit noise in the "agent intention space" rather than just the target space.

3. Joint Distribution Refinement (JDR)

To address the lack of explicit joint dependency modeling in standard diffusion parameterizations (which often factorize across agents), CODA introduces an Energy-Based Model (EBM) formulation.

Mechanism: The final joint distribution is formulated as a product of the diffusion model's marginal distribution and an energy term: $P_{joint} \propto P_{Diff} \cdot \exp(-E_\theta)$ .
Function: The energy term acts as a correction that shifts probability mass toward jointly consistent trajectories while suppressing those that are individually plausible but jointly inconsistent.
Training: The model is trained with a composite loss function including a regression loss, a diversity loss (encouraging distinct modes via KL divergence), and the EBM loss (enforcing pairwise interaction consistency).

Key Contributions

Novel Framework: The proposal of CODA, which integrates rich historical context into the generative process to improve prediction diversity and expressiveness.
Energy-Based Refinement: The introduction of a simple yet effective energy-based formulation for joint trajectory refinement. This approach preserves the plausibility of individual trajectories while significantly improving joint consistency among interacting agents.
Balanced Performance: The demonstration that the framework effectively balances the trade-off between marginal metrics (ADE/FDE) and joint metrics (JADE/JFDE), addressing a common gap in prior joint prediction methods.

Experimental Results

The authors evaluated CODA on four benchmark datasets: ETH/UCY, SDD, NBA, and JRDB.

ETH/UCY: CODA achieved state-of-the-art (SOTA) performance in marginal metrics (Average ADE/FDE of 0.17/0.28), outperforming strong baselines like MoFlow and NRMF. In joint metrics, it remained competitive, achieving the best JADE on the UNIV subset and maintaining stable performance across all scenes.
NBA: CODA demonstrated robust marginal performance, particularly at longer prediction horizons (3.0s and 4.0s), outperforming other methods in ADE/FDE. While NRMF showed slightly better joint metrics (JADE/JFDE), CODA maintained competitive results, indicating effective modeling of complex sports dynamics.
SDD: CODA achieved the best marginal performance (ADE: 6.87, FDE: 10.86), outperforming recent generative approaches. While some methods optimized specifically for joint coherence achieved lower joint metrics, CODA maintained a strong balance, demonstrating effectiveness in highly dynamic environments.
JRDB: CODA showed superior long-term joint consistency, achieving the lowest JADE/JFDE at the 4.8s horizon, suggesting better preservation of global scene coherence in ego-centric environments.

Ablation Studies:
Experiments on the NBA dataset confirmed the distinct roles of the modules:

DCGC and ACIM: Primarily responsible for improving marginal prediction accuracy (ADE/FDE) by capturing richer context.
JDR: Crucial for improving joint trajectory consistency (JADE/JFDE), with minimal impact on marginal metrics when removed.

Significance and Claims

The paper claims that CODA successfully addresses the "diversity-consistency" trade-off in multi-agent motion prediction. By leveraging rich contextual guidance and refining the joint distribution via an energy-based model, the approach generates diverse, realistic trajectories that are also socially consistent.

The authors highlight that their method substantially improves both marginal and joint metrics on ETH/UCY over strong baselines. Furthermore, compared to prior joint prediction methods, CODA delivers significant gains in marginal metrics while maintaining competitive joint performance. The work underscores the importance of modeling diverse yet consistent multi-agent motion, offering a robust solution for safety-critical applications where accurate anticipation of pedestrian and agent behavior is essential.

Limitations Acknowledged:
The authors note that the robust context modeling increases training time. Additionally, while performance is strong, there is still room for improvement in the Collision Rate (CR) metric, which could potentially be alleviated by incorporating LiDAR data. The current pipeline relies solely on coordinate inputs, and integrating richer physical context (e.g., traversable areas) is suggested as a future direction.