Robustness Evaluation of Machine Learning Models for Fault Classification and Localization In Power System Protection

Technical Summary: Robustness Evaluation of Machine Learning Models for Fault Classification and Localization in Power System Protection

Problem Statement
The integration of renewable and distributed energy sources is transforming power systems, introducing dynamic power flows and diverse fault characteristics that challenge conventional protection schemes. While Machine Learning (ML) offers a promising data-driven alternative for centralized Fault Classification (FC) and Fault Localization (FL), its practical deployment is hindered by a lack of robustness evaluation. Existing studies often operate under ideal measurement conditions, failing to address real-world imperfections such as sensor failures, communication losses, and hardware limitations. There is a critical gap in systematically understanding how ML-based protection systems perform when confronted with missing, noisy, or degraded sensor data.

Methodology
The authors propose a unified framework to systematically evaluate the robustness of ML models for FC and FL under limited observability. The methodology consists of four stages:

1. Data Generation: High-fidelity Electromagnetic Transient (EMT) simulations were conducted using DIgSILENT PowerFactory on a benchmark "Double Line" grid topology. The dataset comprises 9,023 simulated fault cases (single-, double-, double-to-ground, and three-phase short circuits) at 90 kV, sampled at 6.4 kHz. Domain randomization was applied to vary line lengths, load conditions, and grid parameters to enhance generalization.
2. Preprocessing: Data was segmented into overlapping 50 ms windows (320 samples) around fault inception, resulting in over 200,000 input tensors representing 48 channels (three-phase currents and voltages from eight protection relays).
3. Model Architecture: A compact Multilayer Perceptron (MLP) with two hidden layers and ReLU activations was employed. The model was trained to perform two tasks: FC (11-class classification) and FL (regression of fault location as a percentage of line length).
4. Robustness Simulation: The framework introduced controlled data degradation scenarios to emulate real-world limitations, including:
   • Sensor Outages: Loss of specific channels (voltage, current, single-phase) or entire relays/substations.
   • Reduced Sampling Rates: Downsampling from 6.4 kHz down to 100 Hz.
   • Temporal Communication Loss: Simulated transient dropouts (5–40 ms) where data segments were zeroed.

Performance was quantified using the F1-score for FC and Mean Absolute Error (MAE) for FL, compared against a baseline trained on complete data.

Key Contributions

• Unified Robustness Framework: The study establishes a consistent methodology for benchmarking ML protection models against realistic degradation scenarios, addressing the fragmentation and lack of comparability in existing literature.
• Systematic Degradation Modeling: It provides a comprehensive analysis of how specific data limitations (e.g., voltage loss vs. current loss, relay outages, sampling reduction) impact model performance.
• Identification of Critical Dependencies: The work identifies specific measurement channels and operating conditions required for resilient operation, moving beyond idealized assumptions to practical constraints.

Results
The evaluation reveals a distinct asymmetry in model robustness between the two tasks:

• Fault Classification (FC): FC models demonstrated high stability. They maintained an F1-score of ~0.99 even under current or voltage loss and remained robust up to a downsampling factor of ×16 (400 Hz). However, FC performance dropped significantly (F1 ~0.85–0.87) when a single phase was missing, indicating a reliance on inter-phase correlations.
• Fault Localization (FL): FL proved significantly more sensitive to data degradation.
  • Voltage Dependency: The loss of voltage measurements caused the most severe degradation, increasing MAE by over 160% (from 7.8% to ~20.6% of line length). Current loss had a moderate impact (71% increase).
  • Spatial Sensitivity: FL accuracy was highly dependent on measurements from central grid locations; the outage of specific substations (e.g., Substation 2) caused error increases of over 100%.
  • Sampling Rate: FL performance degraded gradually with downsampling, with significant errors appearing below 800 Hz.
  • Temporal Loss: Both models showed high resilience to short-term communication losses (5–20 ms), with negligible impact on performance.

Significance and Claims
The paper claims that these findings offer actionable guidance for the robustness-aware design of future ML-assisted protection systems. The results suggest that:

1. Task-Specific Requirements: FC and FL have different data requirements; FC can tolerate lower sampling rates and missing current data, whereas FL critically depends on voltage measurements and central observability.
2. Design Implications: For resilient systems, designers must prioritize voltage redundancy, ensure reliable communication to key substations, and tailor sampling strategies to the specific protection task.
3. Limitations and Future Work: The authors acknowledge that the study relies on a single simulated topology and short-circuit faults. They posit that future work should validate these findings on diverse network topologies, non-fault events, and real-world field conditions, potentially incorporating physics-informed or graph-based models to further improve generalization and robustness.