Probabilistic Reachability Analysis of Multi-scale Voltage Dynamics Using Reinforcement Learning

1. Problem Statement

Modern power systems face increasing complexity due to the integration of renewable energy and inverter-based resources. A critical challenge is assessing voltage stability under multi-scale dynamics and uncertain operating conditions.

• Limitations of Current Methods:

  • Time-Scale Separation: Conventional methods often separate fast (e.g., generator rotor angles, AVRs) and slow (e.g., Load Tap Changers, LTCs) dynamics. This simplification can miss instabilities where slow dynamics trigger fast transient failures.
  • Static Margins: Static stability margins fail to detect instabilities caused by the coupling of long-term and short-term dynamics.
  • Monte Carlo Simulations: While useful for handling uncertainty, Monte Carlo methods are computationally prohibitive for high-dimensional, multi-scale models.
  • Deterministic Reachability: Existing reachability analyses are often deterministic and struggle with the "curse of dimensionality" in nonlinear stochastic systems.

• Core Objective: To develop a scalable framework that quantifies the probabilistic risk of voltage collapse in systems exhibiting coupled fast and slow dynamics under stochastic operating conditions, while identifying the specific instability mechanisms responsible.

2. Methodology

The authors propose a Deep Reinforcement Learning (RL) framework based on Probabilistic Reachability Analysis.

A. Theoretical Foundation: Probabilistic Reachability

The problem is formulated as an invariance-type reachability problem: calculating the probability that system trajectories remain within a "safe set" ($C$) over a finite time horizon $\tau$. The complement of this probability represents the risk of reaching an "unsafe set" ($C^c$).

• State Space Augmentation: To handle the finite horizon and multiplicative nature of safety probabilities, the system state is augmented with a time variable ($h$).
• Absorbing States: Unsafe conditions are modeled as absorbing states. Once the system enters an absorbing state, it remains there.
• Reward Structure: The RL agent receives a reward only if the system remains in the safe set and the time horizon has not expired. This transforms the safety probability calculation into a standard expected-return maximization problem solvable by off-the-shelf RL algorithms (e.g., TD3, DDPG).

B. Mechanism-Specific Multi-Critic Architecture

To address the complexity of multiple instability types (e.g., generator vs. motor instability), the authors introduce a Multi-Critic RL architecture:

1. Instability Detection: Specific criteria are defined to detect different failure modes (e.g., loss of short-term equilibrium, pole slip, motor stalling). These define distinct unsafe sets ($C^c_m$) for each mechanism $m$.
2. Disjoint Absorbing Sets: The overall unsafe set is decomposed into disjoint sets corresponding to specific mechanisms ($S^{(m)}_{abs}$).
3. Multi-Dimensional Rewards: Instead of a single critic, the system employs multiple critics ($Q^{(m)}$), each trained to estimate the risk probability associated with a specific mechanism $m$.
4. Unified Policy Learning: A single shared actor ($\mu$) is optimized to maximize the minimum of the critic values. This ensures the control policy is robust against the worst-case instability mechanism while providing mechanism-specific risk estimates.

3. Key Contributions

1. RL-Based Probabilistic Reachability for Multi-Scale Systems: The paper extends previous RL-based reachability work to handle multiple concurrent instability mechanisms within a unified framework, moving beyond single-metric risk assessment.
2. Mechanism-Specific Risk Quantification: By using a multi-critic architecture, the method can distinguish and quantify the contribution of specific failure modes (e.g., generator OXL limits vs. motor stalling) to the overall voltage collapse risk.
3. First Application to Stochastic Power Systems: To the authors' knowledge, this is the first study to apply probabilistic reachability analysis to power system stability, enabling direct risk quantification under stochastic operating conditions without relying on computationally expensive Monte Carlo simulations.
4. Interpretability: The framework provides interpretable insights into which physical mechanisms dominate the instability risk under different operating scenarios.

4. Numerical Experiments & Results

The framework was validated on a four-bus test system (based on [10]) featuring:

• Components: A remote system, a local generator, an induction motor load, an LTC transformer, and Over-Excitation Limiters (OXL).
• Disturbance: A one-circuit tripping of the tie line, triggering slow LTC action to restore voltage, which eventually stresses the generator.
• Uncertainty: Gaussian perturbations were added to total load demand and motor load ratios.

Key Findings:

• Risk vs. Horizon: The learned risk probability correctly increased with longer time horizons, reflecting the cumulative risk of slow dynamics (LTC action) pushing the system toward instability.
• Mechanism Differentiation:
  • Generator Instability: Risk increased with higher generator active power ($P_g$), consistent with reduced stability margins.
  • Motor Instability: Risk became prominent at high motor load ratios ($R_{motor}$), reflecting the tendency of induction motors to stall under low-voltage conditions.
• Control Effectiveness: The RL agent successfully learned a corrective control policy (adjusting LTC reference voltage) that significantly reduced the risk probability across all time horizons compared to the uncontrolled case.
• Validation: The results aligned with physical expectations (e.g., higher $P_g$ leads to higher risk), demonstrating the method's ability to capture coupled fast-slow dynamics that traditional quasi-static methods might miss.

5. Significance and Conclusion

This paper presents a significant advancement in power system stability assessment by bridging the gap between theoretical reachability analysis and practical, data-driven control.

• Computational Efficiency: It offers a scalable alternative to Monte Carlo simulations for high-dimensional, multi-scale systems.
• Actionable Insights: By decomposing risk into specific mechanisms, grid operators can better understand why a system is unstable (e.g., is it the generator or the motor?) and target controls accordingly.
• Future Outlook: The framework is designed to be extensible to larger systems with high renewable penetration and other instability types (e.g., oscillatory instabilities), paving the way for real-time, probabilistic stability monitoring in modern smart grids.