Distributed Control of Network Systems in the Space of Stabilizing Graph Neural Network Policies

Technical Summary: Distributed Control of Network Systems in the Space of Stabilizing Graph Neural Network Policies

Problem Formulation
The paper addresses the challenge of designing distributed controllers for large-scale, interconnected nonlinear dynamical systems. The objective is to synthesize policies that are simultaneously scalable (generalizing from small to large networks), expressive (capable of learning complex behaviors), and stabilizing (guaranteeing closed-loop stability).

Existing approaches face significant trade-offs:

• Centralized Training with Decentralized Execution (CTDE): While effective in multi-agent reinforcement learning (MARL), these methods often lack formal stability guarantees, suffer from high training variance, and fail to generalize when the number of agents changes between training and deployment.
• Stability-Constrained Learning: Methods that enforce stability via Lyapunov constraints or simple nominal controllers often sacrifice expressivity or rely on restrictive system assumptions.
• Graph Neural Networks (GNNs): While GNNs offer permutation equivariance and robustness to topology changes, using them directly as closed-loop policies can induce instability unless parameters are heavily regularized, limiting their performance.

The authors formulate the problem as a distributed optimal control task over dynamic stochastic policies, subject to the constraint that the induced state and input trajectories must lie in the $\ell_p$ space (ensuring stability) for any disturbance in $\ell_p$.

Methodology
The core contribution is a novel policy parameterization that embeds Graph Neural Networks (GNNs) into a Youla-like magnitude-direction decomposition. This approach guarantees stability by design while retaining the expressivity of neural networks.

1. Magnitude-Direction Decomposition:
   The control input $u_t$ is decomposed into a magnitude term and a direction term:
   $$u_t = |M_t(w_{t:0})| \cdot D_t$$

   • Magnitude ($M_t$): Implemented as a stable operator acting on disturbance feedback ($w$). Specifically, it uses a Linear Recurrent Unit (LRU) acting on the output of a GNN. Stability is enforced by constraining the eigenvalues of the LRU's state transition matrix to be less than 1.
   • Direction ($D_t$): A stochastic term sampled from a distribution conditioned on local observations ($x$). This term is generated by a separate GNN followed by a Recurrent Neural Network (RNN). The direction is bounded ($|D_t| \leq 1$) to ensure the magnitude term dominates the stability properties.

2. Theoretical Guarantees:

   • Stability: The authors prove that if the GNN operator has a finite $\ell_p$-gain and the LRU is stable, the resulting closed-loop system maps disturbances to stable trajectories ($w \to (x, u) \in \ell_p$).
   • Robustness: The paper derives bounds on trajectory deviations resulting from perturbations in the graph communication topology and model parameters. The analysis shows that finite changes in the graph structure or weights do not destabilize the system, even if the policy is untrained.

3. Integration with PPO:
   The authors provide a method to integrate this parameterization with Proximal Policy Optimization (PPO). They derive the log-probability of the control distribution, accounting for the change of variables required by the magnitude-direction decomposition, allowing the use of standard policy-gradient updates.

Key Contributions

• Distributed and Scalable GNN Policies: A Youla-like parameterization that guarantees closed-loop stability while utilizing GNNs. This enables training on small networks and transferring to larger systems with different topologies without retraining.
• Robustness to Perturbations: Theoretical proof that the policy is robust to changes in both the communication graph topology and model parameters, ensuring stability is retained under structural variations.
• PPO Compatibility: A practical recipe for training these structured policies using standard reinforcement learning algorithms.
• Empirical Validation: Demonstration that the method generalizes to unseen system sizes and topologies, outperforming ablated versions that lack stability guarantees or distributed structure.

Experimental Results
The authors evaluate their method on a Multi-Agent Particle Environment involving agents with nonlinear collision and speed dynamics. The task is to navigate to target positions while avoiding collisions.

• Stability: In untrained scenarios, the proposed method drives the system to equilibrium, whereas purely GNN-based policies (without the magnitude-direction structure) result in unstable or oscillatory behavior.
• Generalization: Policies trained on 5 agents successfully generalize to 7 and 10 agents. In contrast, centralized MLP actors fail to scale due to rigid input size requirements.
• Ablation Studies:
  • Removing the stabilizing base policy (Ablation 1) leads to slower convergence and lower rewards.
  • Replacing the GNN actor with a centralized MLP (Ablation 2) prevents scaling to larger agent counts.
  • Replacing the GNN critic with an MLP (Ablation 3) degrades performance, highlighting the benefit of graph-structured information in the learning process.

Significance and Claims
The paper claims to provide a unifying framework that simultaneously achieves distributed computation with guaranteed closed-loop stability, robustness to topology changes, and state-of-the-art performance. The authors emphasize that their approach allows for the "synthesis of scalable and generalizable controllers" that are "expressive and closed-loop stable by design."

The significance lies in bridging the gap between the theoretical guarantees of Youla parameterization and the practical scalability of GNNs. By decoupling stability (handled by the magnitude term) from expressivity (handled by the direction term), the method avoids the conservatism typically associated with stability-constrained learning. The authors modestly note that while the theoretical bounds on trajectory deviations are not tight (due to the stochasticity of the direction term), the practical benefits of stability and generalization outweigh this limitation. Future work is suggested to include probabilistic analysis of trajectory deviation bounds and extension to output-feedback systems.