Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication

1. Problem Statement

The paper addresses the challenge of cooperative Multi-Agent Reinforcement Learning (MARL) under partial observability (Dec-POMDP). In such settings, agents must coordinate to achieve shared goals but only possess local observations.

• The Core Issue: Existing methods typically optimize communication protocols for intermediate objectives (e.g., message reconstruction accuracy or mutual information) rather than downstream decision quality. This leads to an "objective mismatch" where agents exchange information that is statistically accurate but irrelevant to the team's actual performance.
• The Gap: Standard Decision-Focused Learning (DFL), which trains predictors to optimize downstream decisions, has largely been restricted to single-agent settings with exogenous uncertainty. It has not been extended to multi-agent systems where communication creates endogenous uncertainty (i.e., messages actively change how other agents act, creating feedback loops).

2. Methodology: SeqComm-DFL Framework

The authors propose SeqComm-DFL, a framework that unifies sequential communication, model-based RL, and decision-focused learning. The core philosophy is to treat the communication module as a "predictor" and the MARL policy selection as an "optimization task," optimizing the former directly for the latter's success.

A. Value-Aware Message Generation

Instead of minimizing reconstruction error, the communication module is trained to maximize the decision quality improvement ($\Delta Q$) for receiving agents.

• Metric: $\Delta Q_j(m_i) = \max_a Q_j(o_j, m_i, a) - \max_a Q_j(o_j, \emptyset, a)$. This measures how much agent $j$'s optimal Q-value improves upon receiving message $m_i$ from agent $i$ compared to receiving no message.
• Loss Function: A value-aware auxiliary loss ($L_{VA}$) is introduced to maximize the sum of $\Delta Q$ across all receivers.
• Refinement: Messages are refined based on estimated decision impact. To handle the "biased critic" problem during early training, the method uses Monte Carlo rollouts to ground $\Delta Q$ estimates before transitioning to critic-based estimates.

B. Sequential Stackelberg Conditioning

To resolve coordination ambiguity and the "relative overgeneralization" problem, agents do not act simultaneously. Instead, they act in a sequential Stackelberg order (Leader-Follower structure).

• Guidance Potential (GP): Agents are prioritized based on their capacity to improve team outcomes. $GP_i$ measures the value of agent $i$ acting as a leader versus a follower. Agents with high private information (large coordination information gap) are prioritized to lead.
• Mechanism: The leader commits to an action based on local info; followers condition their actions on the leader's message and action. This breaks symmetry and allows followers to reach Pareto-superior equilibria.
• Counterfactual Influence: A regularization loss ($L_{inf}$) ensures messages causally change receiver behavior (measured via KL divergence), preventing agents from ignoring messages.

C. Decision-Focused Bilevel Optimization

The framework extends Optimal Model Design (OMD) to communication-augmented world models.

• Bilevel Formulation:
  • Inner Loop: Trains a critic $Q_w$ on the world model's predictions ($f_\theta$).
  • Outer Loop: Updates the world model parameters $\theta$ to minimize the true environment loss, assuming the critic is optimally trained on the model.
• Implicit Differentiation: To compute the gradient of the outer loss with respect to $\theta$ (the hypergradient), the method uses the Implicit Function Theorem (IFT). This avoids backpropagating through thousands of inner-loop gradient steps, which is computationally expensive and prone to vanishing gradients.
• Efficiency: The inverse Hessian-vector product required by IFT is computed efficiently using the Conjugate Gradient (CG) method.
• Preventing Apathy: An auxiliary "message-awareness" loss is added to the inner loop to prevent the critic from learning to ignore messages (treating them as noise).

3. Key Contributions

1. First Multi-Agent DFL Framework: Extends Decision-Focused Learning to multi-agent systems with endogenous, communication-induced uncertainty.
2. Value-Aware Communication: Introduces a training paradigm where messages are optimized strictly for their impact on receiver decision quality ($\Delta Q$), not information reconstruction.
3. Sequential Stackelberg Conditioning: Proposes a dynamic priority ordering (Guidance Potential) that allows agents to act sequentially, resolving coordination gaps inaccessible to simultaneous action selection.
4. Theoretical Guarantees:
   • Information-Theoretic Bounds: Proves that communication value scales with the coordination information gap and action variance.
   • Convergence: Establishes $O(1/\sqrt{T})$ convergence for the bilevel optimization problem.
5. Efficient Training: Adapts Optimal Model Design with QMIX factorization and implicit differentiation for scalable, end-to-end training.

4. Experimental Results

The method was evaluated on two benchmarks: a collaborative Hospital Environment and the StarCraft Multi-Agent Challenge (SMAC).

• Hospital Environment (Dec-POMDP with Information Asymmetry):
  • Setup: 3 specialists (Cardiology, Pulmonology, Neurology) treating patients with hidden risks specific to their specialty.
  • Result: SeqComm-DFL achieved 4–6x higher cumulative rewards compared to the Optimal Model Design (OMD) baseline. It maintained a severity improvement of 0.2 vs. 0.05 for the baseline, demonstrating effective avoidance of "blind treatment" penalties.
• SMAC Benchmarks:
  • Setup: Various maps (2s_vs_1sc, 3s_vs_5z, 2s3z, 8m).
  • Result: SeqComm-DFL achieved the highest win rates across all maps, improving by 13–15 percentage points over the original SeqComm and significantly outperforming OMD.
• Ablation Studies: Removing the value-aware loss ($L_{VA}$) reduced performance by ~12%, and removing sequential conditioning reduced it by ~7%, confirming the necessity of each component.

5. Significance and Impact

• Paradigm Shift: Moves communication design from "information maximization" (sending everything) to "decision optimization" (sending only what improves the team's outcome).
• Real-World Applicability: The framework is particularly relevant for bandwidth-constrained, safety-critical domains like collaborative healthcare (specialist coordination) and autonomous systems, where information asymmetry is inherent and decision quality is paramount.
• Theoretical Rigor: Provides formal bounds on why communication is necessary in partial observability and guarantees convergence for complex bilevel multi-agent training.

In summary, SeqComm-DFL provides a principled, theoretically grounded, and empirically superior approach to learning communication protocols in multi-agent systems, ensuring that agents share information strategically to maximize collective decision quality.