Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

Technical Summary: Decentralized Diffusion Policy Learning for Enhanced Exploration in Cooperative Multi-agent Reinforcement Learning

Problem Statement
Cooperative Multi-Agent Reinforcement Learning (MARL) requires agents to explore complex joint action spaces to discover high-reward coordination patterns. A prominent class of algorithms for this setting, Decentralized Softmax Policy Gradient (DecSPG), utilizes energy-based policy updates that naturally generate multimodal action distributions, facilitating diverse exploration. However, in practice, exact energy-based policies are intractable to maintain, particularly in continuous action spaces. Consequently, existing methods project these policies onto the Gaussian policy class for computational efficiency.

The authors identify a critical limitation in this approach: Gaussian policies are unimodal and lack the expressiveness required to capture multimodal action distributions. This restriction severely hinders exploration, causing algorithms to converge prematurely to suboptimal equilibria. The paper argues that this failure mode is particularly severe in multi-agent settings, where the number of viable coordination patterns grows with the number of agents, and the cost of missing these patterns compounds exponentially.

Methodology
To address the expressiveness bottleneck, the paper proposes Decentralized Diffusion Policy Learning (DDPL). This framework parameterizes each agent's decentralized policy using a Denoising Diffusion Probabilistic Model (DDPM). DDPMs serve as expressive generative models capable of representing complex, multimodal distributions while remaining efficient to train and sample from.

A central challenge in deploying diffusion policies within the online MARL setting is the lack of efficient training algorithms. Standard score matching methods typically require samples from the target distribution, which are unavailable during online training before the policy update. To overcome this, the authors introduce Importance Sampling Score Matching (ISSM).

• ISSM Mechanism: Instead of requiring samples from the target energy-based distribution $\pi^{\star,k}_i$, ISSM draws samples from an easy-to-sample distribution (specifically, the previous policy $\pi^{k-1}_i$). It then minimizes a reweighted score matching loss where the weights are determined by the importance sampling ratio between the target and sampling distributions.
• Theoretical Guarantee: The paper derives a theoretical guarantee for ISSM, showing that the learned policy converges to the target distribution with a sample complexity that depends on the fourth-order Rényi divergence between the target and sampling distributions. As the policy updates progress and the current policy approaches the target, the required sample size for accurate updates decreases.
• Algorithm: The full DDPL algorithm (Algorithm 2) integrates these components into a decentralized framework. Agents maintain local Q-networks and diffusion policies. In each epoch, agents estimate the averaged Q-function and update their diffusion policies via a single gradient step on the ISSM loss, using the previous policy as the sampling distribution.

Key Contributions

1. Identification of a Failure Mode: The authors formally demonstrate that projecting energy-based policies onto Gaussian classes creates a specific failure mode in cooperative MARL. Through a constructed single-agent MDP and a multi-agent identical-interest game, they prove that Gaussian approximations can cause the success probability of finding optimal equilibria to decay exponentially with the number of agents (specifically bounded by $0.45^n$).
2. DDPL Framework: They propose DDPL, a DecSPG-based algorithm that utilizes DDPMs to parameterize decentralized policies, enabling the representation of multimodal action distributions essential for effective exploration.
3. Online Training via ISSM: The paper develops a novel Importance Sampling Score Matching objective that allows for the efficient online training of diffusion policies without requiring samples from the target distribution. This method comes with theoretical sample-complexity guarantees and Nash Equilibrium (NE) gap bounds.
4. Empirical Validation: The method is evaluated on representative continuous-action MARL benchmarks, including Multi-Agent Particle Environment (MPE), Multi-Agent MuJoCo (MaMuJoCo), IsaacLab, and a JAX-reimplemented StarCraft Multi-Agent Challenge (SMAX).

Results
Experimental results indicate that DDPL consistently outperforms state-of-the-art baselines (HAPPO, MAPPO, and HASAC) across 8 continuous action tasks.

• Performance: DDPL achieved the highest mean return on 7 out of 8 tasks. Notable improvements include a 25.78% increase on the IsaacLab ShadowHandOver task, a 132.56% increase on the 3s_vs_5z StarCraft task, and a 155.84% increase on the 6h_vs_8z task compared to the best baseline.
• Exploration Dynamics: Analysis of learning curves suggests that DDPL's gains stem from sufficient exploration. In tasks like 3s_vs_5z, DDPL initially lags behind baselines due to broader exploration but rapidly converges to near-optimal returns, whereas Gaussian-based baselines get trapped in suboptimal equilibria.
• Ablation: Comparisons between DDPL with diffusion policies and a variant using Gaussian policies confirm that the multimodal nature of diffusion policies is critical. The Gaussian variant converges early to a suboptimal equilibrium (return $\approx 0.65$), while the diffusion variant discovers higher-reward modes and converges to a superior equilibrium (return $\approx 2.0$).
• Efficiency: The training overhead of DDPL is reported to be 39% smaller than DACER and 81% faster than QVPO, attributed to the efficiency of the ISSM online training method. Inference time remains moderate (below 1 ms).

Significance and Claims
The paper claims that the limited expressiveness of standard policy parameterizations (specifically Gaussian) is a primary obstacle to effective exploration in cooperative MARL. By leveraging diffusion models, the authors demonstrate that it is possible to maintain the theoretical benefits of energy-based policy updates (multimodality and convergence to Nash equilibria) while ensuring tractability for online training.

The work highlights that in multi-agent settings, the cost of insufficient exploration compounds with the number of agents. DDPL offers a solution that scales better in terms of exploration capability, allowing agents to discover coordination patterns that unimodal policies miss. The authors position DDPL as a significant step toward enabling more robust and effective cooperative learning in complex, continuous-action environments.

Limitations
The authors acknowledge that DDPL currently parameterizes fully decentralized policies and does not exploit network MARL settings where low-cost communication between neighboring agents is encouraged. Additionally, the framework is currently focused on identical-interest settings, with general-sum or competitive settings left for future work.