Topology-Aware Graph Reinforcement Learning for Energy Storage Systems Optimal Dispatch in Distribution Networks

1. Problem Statement

The optimal dispatch of Energy Storage Systems (ESS) in distribution networks (DNs) is a complex, constrained optimization problem. It requires jointly minimizing operational costs (electricity purchasing) and maintaining voltage security under time-varying conditions (load, generation, prices) and potential topology changes (switching operations).

• Challenges:
  • Computational Latency: Traditional model-based methods (e.g., Non-Linear Programming - NLP) are computationally expensive when solved in real-time, especially for large networks or frequent topology changes.
  • Topology Dependence: The impact of an ESS action on system voltages is governed by electrical connectivity and line impedances. Standard Reinforcement Learning (RL) agents using flattened state vectors often fail to capture these spatial dependencies, leading to poor generalization when the network topology changes.
  • Generalization: Existing RL approaches struggle to transfer policies across networks of different sizes or reconfigured topologies without retraining.

2. Methodology

The authors propose a Topology-Aware Graph Reinforcement Learning (GNN-RL) framework based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm.

A. Mathematical Formulation

The problem is formulated as a Markov Decision Process (MDP):

• State ($s_t$): Includes time step, electricity price, active power demand, ESS State of Charge (SOC), and node voltages (specifically at ESS nodes).
• Action ($a_t$): Charging/discharging power for each ESS.
• Reward ($r_t$): A weighted sum of two components:
  1. Economic: Negative cost of electricity purchased (reward for discharging during high prices).
  2. Technical: Penalty for voltage deviations outside allowable limits ($[V_{min}, V_{max}]$).

B. GNN-Based Architecture

Instead of using standard Multi-Layer Perceptrons (MLPs) with flattened inputs, the authors embed a Graph Neural Network (GNN) encoder into the actor-critic structure:

• Graph Representation: The distribution network is modeled as a graph $G=(E, X)$, where $E$ is the adjacency matrix (connectivity) and $X$ is the node feature matrix.
• Node Features: Include time, price, demand, SOC (zero-padded for non-ESS nodes), voltage (zero-padded for non-ESS nodes), and a binary indicator for ESS presence.
• Asymmetric Actor-Critic Design:
  • Actor: Uses graph embeddings derived only from ESS nodes ($G_B$) to generate actions. This focuses the policy on storage-specific features while leveraging global topology context.
  • Critic: Uses global mean pooling to aggregate embeddings from all nodes ($G'$), providing a system-wide view of the state and constraints for value estimation.

C. GNN Variants Evaluated

Three specific graph convolutional architectures were integrated and compared:

1. GCN (Graph Convolutional Network): Uses normalized adjacency for stable, symmetric neighbor aggregation.
2. TAGConv (Topology Adaptive Graph Convolution): Uses $K$-order polynomial filters to capture multi-hop dependencies in a single layer, avoiding deep stacking and over-smoothing.
3. GAT (Graph Attention Network): Uses an attention mechanism to dynamically weight neighbors, allowing the model to focus on electrically influential nodes.

3. Key Contributions

1. Topology-Aware Encoder: Development of a GNN-based feature encoder that explicitly incorporates network connectivity and electrical coupling into the RL state representation.
2. Asymmetric Actor-Critic Framework: Integration of the GNN encoder into TD3, where the actor focuses on local ESS actions and the critic evaluates global system constraints.
3. Systematic Evaluation: Comprehensive benchmarking of GCN, TAGConv, and GAT against a standard NN baseline and a global NLP optimizer on two distinct systems (IEEE 34-bus and 69-bus).
4. Generalization Analysis: Rigorous testing of robustness under:
   • In-network reconfiguration: Changing branch connections within the same system.
   • Cross-system transfer: Transferring policies between systems of different sizes (34-bus $\leftrightarrow$ 69-bus).

4. Experimental Results

A. Performance on Training Topologies

• Execution Speed: RL agents are 86x to 614x faster than solving NLP online. The speedup is more significant for the larger 69-bus system.
• Economic Performance:
  • On the 34-bus system, the standard NN baseline performed best, with GNN variants achieving 92–99% of the NN's cost savings.
  • On the 69-bus system (more complex), TD3-GCN and TD3-TAGConv outperformed the NN baseline, achieving ~109% and ~102% of the NN's saved costs, respectively.
• Voltage Regulation: GNN-based controllers consistently reduced the frequency and magnitude of voltage violations compared to the NN baseline. This advantage was most pronounced on the 69-bus system and under reconfiguration.
• GNN Variant Comparison:
  • TAGConv and GAT showed strong representational capacity but GAT exhibited higher sensitivity to random seeds (optimization instability).
  • TAGConv offered a good balance, capturing multi-hop dependencies effectively.

B. Generalization and Transfer

• In-Network Reconfiguration: Policies trained on the baseline topology generalized well to reconfigured topologies (minor performance drop). GNNs maintained more consistent constraint satisfaction (voltage) than NNs when electrical coupling changed.
• Cross-System Transfer (Zero-Shot):
  • Transferring policies between fundamentally different systems (34 $\to$ 69 or 69 $\to$ 34) resulted in significant performance degradation.
  • Economic accuracy dropped to ~30–50%, and voltage violations increased.
  • Asymmetry: Policies trained on the larger (69-bus) system transferred slightly better to the smaller system than vice versa, suggesting that complex systems learn richer representations of network interactions.

5. Significance and Conclusion

• Practical Viability: The proposed framework demonstrates that GNN-based RL can provide near-instant, high-quality dispatch decisions, overcoming the latency bottleneck of traditional optimization.
• Topology Awareness: Explicitly modeling the grid as a graph is crucial for voltage regulation, especially in larger, deeper radial networks where local actions have non-local effects.
• Limitations & Future Work: While GNNs excel at in-network reconfiguration, zero-shot transfer across different system scales remains challenging. The authors suggest that future work should focus on topology-diversified training and lightweight adaptation (fine-tuning) to enable reliable cross-system deployment.

Summary: This paper establishes that integrating Graph Neural Networks into Reinforcement Learning significantly improves the robustness and voltage regulation capabilities of ESS dispatch controllers in distribution networks, particularly as network complexity increases, though cross-system generalization requires further adaptation strategies.