Technical Summary: Prism – Spectral Parameter Sharing for Multi-Agent Reinforcement Learning

Problem Statement

Multi-Agent Reinforcement Learning (MARL) faces a fundamental trade-off between scalability and policy heterogeneity. While parameter sharing is a standard strategy to reduce memory overhead and improve sample efficiency by having all agents learn from a single set of weights, fully shared architectures often collapse into homogeneous behaviors, failing to capture necessary agent differences. Conversely, methods that induce diversity—such as clustering agents or applying node/edge-level pruning and masking—often compromise resource efficiency or rely heavily on initial data distributions, making them sensitive to non-stationary environments. Existing approaches struggle to simultaneously achieve high performance, memory efficiency, and robust agent differentiation.

Methodology: The Prism Framework

The authors propose Prism, a parameter sharing framework that operates in the spectral domain rather than the weight space. Instead of sharing raw weights or masking specific nodes/edges, Prism decomposes shared network weights using Singular Value Decomposition (SVD) and induces diversity through learnable spectral masks.

Core Mechanism

1. SVD Parameterization:
   The shared weight matrix $W \in \mathbb{R}^{d \times k}$ is decomposed into $W = U \Sigma V^\top$.

   • Shared Components: The left singular vectors ($U$) and right singular vectors ($V$) are shared across all agents, providing a common basis for representation.
   • Agent-Specific Components: The singular values ($\Sigma$) are adapted per agent.

2. Spectral Space Masking:
   Each agent $i$ learns a specific mask $m_i$ applied to the singular values. The agent-specific singular values $\sigma_i$ are constructed as:
   $$\sigma_i = \text{concat}(\sigma_{\text{common}}, \sigma_{\text{separate}} \odot m_i)$$
   where $\odot$ denotes the Hadamard product. This allows agents to selectively activate or scale specific spectral modes, effectively allocating distinct subspaces within the shared network without introducing redundant parameters.

3. Regularization:
   To ensure stable training and meaningful diversity, Prism employs two regularization terms:

   • Diversity Regularization: Penalizes the similarity between agents' spectral masks (using a Straight-Through Estimator for binary masks) to encourage agents to attend to different spectral components.
   • Orthogonal Regularization: Enforces orthogonality on the shared singular vector matrices ($U$ and $V$) to prevent the collapse of the SVD decomposition and reduce interference between basis components.

Key Contributions

1. Spectral Parameter Sharing: A novel framework that induces agent differentiation by allocating distinct spectral subspaces within a shared network, moving beyond traditional weight-space masking.
2. Resolution of Scalability-Heterogeneity Trade-off: By inducing diversity solely in the spectral space, Prism avoids the redundant parameters associated with node/edge-level masking, significantly improving resource efficiency while maintaining policy heterogeneity.
3. Empirical Validation: Extensive experiments demonstrate that Prism achieves competitive or superior performance across diverse benchmarks while substantially reducing memory overhead compared to state-of-the-art methods.

Experimental Results

The authors evaluated Prism on both homogeneous (LBF, SMACv2) and heterogeneous (MaMuJoCo) environments, comparing it against baselines including No Parameter Sharing (NoPS), Full Parameter Sharing (FuPS), Clustering (SePS, AdaPS), and Structured Pruning/Masking (SNP, Kaleidoscope).

• Performance:
  • In homogeneous settings (LBF, SMACv2), Prism achieved performance comparable to or better than fully shared methods (FuPS), outperforming clustering-based approaches that struggled with stochasticity.
  • In heterogeneous settings (MaMuJoCo), Prism consistently outperformed FuPS and showed robust performance across tasks where other methods (like AdaPS) varied significantly. It notably surpassed Kaleidoscope, which applies masks to critic networks.
• Resource Efficiency:
  • Under parameter budget constraints (50% of NoPS parameters), Prism maintained stable performance, whereas other methods degraded more significantly.
  • Scalability: As the number of agents increased, Prism exhibited the lowest growth in resource overhead. Unlike edge-level masking (Kaleidoscope) or node-level masking (SNP), where overhead scales linearly or super-linearly with agent count, Prism's spectral masks remain compact, resulting in a significantly lower normalized resource overhead.
• Ablation Studies:
  • Diversity Regularization: Crucial for heterogeneous environments; removing it caused significant performance drops in MaMuJoCo.
  • Orthogonal Regularization: Essential for training stability under SVD factorization.
  • Sharing Ratio: Higher shared spectral ratios benefited homogeneous tasks (improving convergence), while heterogeneous tasks relied more on diversity regularization than the sharing ratio itself.

Significance and Claims

The paper claims that Prism effectively addresses the scalability challenge in MARL by decoupling the need for parameter sharing from the need for policy homogeneity. By leveraging the spectral domain, the method allows agents to specialize without the memory cost of maintaining separate subnetworks or complex masking structures.

The authors highlight that Prism is particularly significant for large-scale multi-agent systems where memory efficiency and the ability to handle agent heterogeneity are critical. While the method introduces hyperparameters for tuning the ratio of shared to specific components, the authors note that the approach is generally robust to these settings. The paper concludes that spectral parameter sharing offers a promising direction for achieving both memory efficiency and behavioral diversity, though future work could explore extending these principles to on-policy algorithms.