🤖 machine learning

CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

CODA is an algorithm-agnostic augmentation module that enables multi-agent offline reinforcement learning to achieve better coordination by using a diffusion-based trajectory generator to produce synthetic, on-policy experience that evolves alongside the agents' changing behaviors.

Original authors: Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek, Anya Sims, Riccardo Zamboni, Trevor McInroe, John Torr, Elliot Fosong

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Marcel Hedman, Kale-ab Abebe Tessera, Juan Claude Formanek, Anya Sims, Riccardo Zamboni, Trevor McInroe, John Torr, Elliot Fosong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a group of dancers a complex, synchronized routine, but there is a catch: you aren't allowed to be in the room while they practice.

Instead, you have to give them a video of a previous performance and say, "Watch this, and try to do better."

This is the core problem the researchers in this paper are solving. In the world of AI, this is called Offline Multi-Agent Reinforcement Learning.

The Problem: The "Frozen Video" Trap

In "Online" learning, agents (the dancers) learn by interacting. If Dancer A moves left, Dancer B sees it and adjusts. They "co-adapt"—they evolve together in real-time.

In "Offline" learning, the agents are stuck looking at a "frozen video" (a static dataset). Because the video never changes, the dancers can't react to each other's improvements. If Dancer A decides to try a new, better move, they can't see if Dancer B is ready for it. They end up "miscoordinating"—one person tries to do a leap while the other tries to do a slide, and the whole performance falls apart.

The researchers call this Coordination Failure. Even if the video shows great dancing, the agents can't figure out how to improve together because they are essentially practicing in isolation, looking at a past that no longer reflects who they are becoming.

The Solution: CODA (The "Dream Simulator")

The researchers created CODA. Think of CODA not as a video, but as a high-tech Dream Simulator.

Instead of just re-watching the old video, CODA uses a technology called Diffusion (the same tech used to create AI art like Midjourney) to "dream up" new practice sessions.

But here is the genius part: These dreams are "On-Policy."

This means the simulator doesn't just dream of random dancing; it asks the dancers, "Based on how you are currently practicing, what would a successful routine look like if you both improved slightly?"

The Analogy: The Adaptive Mirror
Imagine the dancers are practicing in front of a magical, smart mirror.

Old Way: The dancers look at a photograph of yesterday's rehearsal. They can't learn from their new movements.
CODA Way: The dancers look into a magical mirror that shows them a "simulated future." As the dancers get better, the mirror's reflection updates instantly. If Dancer A starts moving faster, the mirror shows a version of Dancer B that is also moving faster to keep up.

By "dreaming" these updated scenarios, CODA provides the dancers with the missing feedback loop. It allows them to practice "co-adaptation" in their heads before they ever step onto the real stage.

Does it work?

The researchers tested CODA in two main ways:

Simple Math Games: In games where agents had to coordinate to get a high score (like a multiplication game), the old methods failed and got stuck. CODA allowed them to find the perfect "rhythm" and reach the highest score.
Complex Robot Simulations (MaMuJoCo): They tested it on complex, multi-robot tasks. CODA proved it could "steer" the simulated practice sessions toward much better, more expert-level movements.

In Short:

CODA takes the "static" problem of offline learning and makes it "dynamic" again. It uses AI to generate "what-if" scenarios that evolve alongside the agents, allowing them to learn how to work as a team, even when they can't actually interact with the real world.

Technical Summary: CODA (Coordination via On-Policy Diffusion for Multi-Agent Offline RL)

CODA is a novel data augmentation framework designed to solve the fundamental problem of coordination failure in cooperative offline Multi-Agent Reinforcement Learning (MARL).

1. The Problem: Offline Coordination Failure

In offline MARL, agents must learn optimal joint behaviors from a fixed, static dataset $\mathcal{D}_{off}$ collected by a behavior policy $\pi_{off}$ . The authors identify a critical structural flaw in standard "Best Response under Data" (BRUD) algorithms (like MADDPG or MASAC):

Lack of Co-adaptation: In online RL, agents adapt to one another because their policy updates change the distribution of future experiences (an endogenous feedback loop). In offline RL, this loop is broken.
The Mismatch: When an agent updates its policy $\pi_i$ , it optimizes against teammate actions sampled from the static dataset $\mathcal{D}_{off}$ rather than the teammates' current evolving policies.
Consequence: This leads to "miscoordination," where agents converge to suboptimal, incompatible behaviors. Even with high dataset coverage, agents fail to "agree" on a joint strategy because they are effectively learning against "stale" versions of their teammates.

2. Methodology: Restoring Policy Evolution

CODA restores the missing feedback loop by using Diffusion Models to generate synthetic trajectories that are conditioned on the agents' current joint policies.

Key Technical Components:

Trajectory-Level Diffusion: Instead of modeling single actions, CODA learns a diffusion prior over entire joint trajectories $\tau = (s_0, o_0, a_0, \dots, s_H)$ . This allows the model to capture complex temporal and inter-agent correlations.
On-Policy Guidance: To make the generated data "on-policy," CODA uses two conditioning mechanisms to tilt the offline distribution toward the current joint policy $\pi_{curr}$ $π_{c u r r}$ :
1. Classifier-Free Guidance (CFG): If the policy can be represented as a compact embedding, the diffusion model is trained to condition directly on that embedding.
2. Policy-Guided Sampling (Implicit): If no compact embedding exists, CODA uses the gradient of the current policy's log-likelihood ( $\nabla \log \pi_{curr}$ ) to nudge the denoising process. This effectively samples from a "policy-tilted" distribution: $\tilde{p}_{\pi_{curr}}(\tau) \propto p_{\pi_{off}}(\tau) \exp(\lambda \sum \log \pi_{curr}(a|o))$ .
Algorithm Agnostic: CODA acts as a "plug-and-play" augmentation module. It generates synthetic trajectories which are then mixed with the original offline data to train any existing offline MARL algorithm.

3. Key Contributions

Identification of the "Co-adaptation Gap": Formally demonstrates that offline coordination failure is not just a coverage issue, but a structural failure of the learning dynamics due to the absence of endogenous policy evolution.
On-Policy Diffusion Framework: Introduces a method to simulate on-policy experience using only offline data by combining trajectory diffusion with policy-based guidance.
Theoretical Insight: Provides mathematical proofs (via the Multiplication and Twin Peaks games) showing how standard BRUD updates result in "constant" or "non-adaptive" gradient fields that ignore teammate evolution.

4. Empirical Results

The authors evaluated CODA across two distinct benchmarks:

Polynomial Games (Coordination Focus): In games like "Multiplication" and "Twin Peaks," standard MADDPG fails to coordinate and converges to suboptimal points (e.g., the origin). CODA successfully resolves these pathologies, allowing agents to follow the correct joint gradient and reach the global optimum.
MaMuJoCo (Complex Control Focus): In high-dimensional continuous control tasks, CODA demonstrates superior performance on "Replay" and "Good" datasets. It proves that the guidance scale $\lambda$ can effectively "steer" generated trajectories to match the current policy's distribution.

5. Significance

CODA represents a significant step toward making offline MARL viable for real-world applications (like robotics or resource management) where interactive exploration is impossible. By bridging the gap between offline data stability (staying near the data manifold) and online co-adaptation (evolving with teammates), it provides a scalable mechanism for training coordinated multi-agent systems without requiring new environment interactions.