⚡ electrical engineering

Identification and Visualization of Correlation Structures in Large-Scale Power Quality Data

This paper presents an enhanced methodology for automatically analyzing and visualizing correlation structures in large-scale power quality datasets by adapting existing frameworks for shorter observation periods and employing hierarchical clustering and multidimensional scaling on aggregated Spearman correlation coefficients, as demonstrated with data from 85 sites in the German transmission system.

Original authors: Max Domagk, Jan Meyer, Marco Lindner

Published 2026-03-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Max Domagk, Jan Meyer, Marco Lindner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the electrical grid as a massive, bustling city. In this city, the electricity flowing through the wires is like the traffic. Sometimes, the traffic is smooth and steady; other times, it gets bumpy, jerky, or full of strange "potholes" (which engineers call Power Quality issues).

To keep the city running safely, engineers place thousands of "traffic cameras" (measurement sites) all over the grid. These cameras record data 24/7, generating a mountain of information so huge that looking at it one by one is like trying to read every single book in a library to find a specific fact. It's overwhelming.

This paper presents a new way to make sense of that mountain of data. Here is the story of how they did it, explained simply:

1. The Problem: Too Many Voices, Too Much Noise

The researchers had data from 85 different locations in Germany's high-voltage grid. At each spot, they were tracking 20 different "moods" of the electricity (like how steady the voltage is, how much "flicker" there is, or how "distorted" the waves are).

If you tried to look at how these 20 moods relate to each other at 85 different spots, you'd have thousands of possible connections to check. It's like trying to figure out who is friends with whom in a stadium full of 10,000 people just by looking at a spreadsheet. You'd miss the patterns.

2. The Solution: The "Daily Diary" Approach

Instead of looking at the whole year of data as one giant, messy block, the researchers broke it down into daily chapters.

The Analogy: Imagine trying to understand if two people are friends. If you look at their entire lives, they might seem unrelated because they lived in different cities for a decade. But if you look at their daily interactions for a month, you might see they text every morning.
The Method: They calculated how closely related two measurements were every single day. Some days, the connection was strong; other days, it was weak. By averaging these daily "friendships" using a special math trick (Fisher's z-transformation), they found the true, long-term relationship between the variables, ignoring the daily noise.

3. The Filter: Finding the "Super-Friends"

Once they had the average relationships, they needed to filter out the weak ones.

The Analogy: Imagine a party where everyone is talking. You only care about the people who are shouting to each other over the music. The researchers set a "volume threshold." If two variables (like the 5th harmonic and total distortion) were "shouting" (highly correlated) at a site, they kept that connection. If they were just whispering, they ignored it.
The Result: They created a "Consistency Map." This map showed which relationships happened often across many different sites, rather than just by chance at one spot.

4. The Visualization: Turning Data into a Map

Now they had a list of "super-friends," but it was still hard to read. So, they used two creative tools to turn the data into a picture:

Hierarchical Clustering (The Family Tree):
Imagine sorting a huge pile of mixed-up socks. You group the red ones together, then the blue ones, then the striped ones. The researchers did this with electricity data. They built a "family tree" (dendrogram) showing which electrical problems tend to happen together.
- What they found: The "low-order" harmonics (like the 3rd, 5th, and 7th) and "flicker" were like a tight-knit family that always stuck together. The "high-order" harmonics (the 9th, 11th, 15th) were the loners, rarely correlating with anyone.
Multidimensional Scaling (MDS) (The Social Distance Map):
Imagine a map of a city where the distance between two buildings represents how "close" they are in friendship, not how far apart they are geographically.
- The Result: On this map, the "flicker" and "5th harmonic" were sitting right next to each other on a park bench. The "9th harmonic" was way over on the other side of the city, alone. This visual map made it instantly obvious which problems travel together through the grid.

5. The Big Discovery: Who is Connected to Whom?

Finally, they applied this to the 85 different locations to see if the problems traveled across the country.

The "Local" vs. "Global" Effect:
They found that some problems, like the 3rd voltage harmonic and flicker, were "global citizens." They showed up at almost every location in the grid, often at the same time. It's like a weather front moving across the whole country; if it's windy in Berlin, it's windy in Munich.
The "Local" Problems:
Other problems were "local residents." They only appeared in specific neighborhoods (specific voltage levels or specific substations) and didn't seem to care about what was happening next door.

Why Does This Matter?

This method is like giving the grid operators a smart radar.

Root Cause Analysis: If a weird spike happens, they can look at the map and say, "Ah, the 5th harmonic is acting up, which means the 7th harmonic will probably follow, and this is likely affecting the whole 380kV network."
Saving Money: They realized that some measurement sites are so similar (so "best friends") that they are redundant. You don't need cameras in two places if they are always watching the exact same thing. You can turn off the extra cameras and save money.

In short: The paper teaches us how to stop drowning in data and start seeing the "social network" of electricity, revealing which problems travel together and which ones stay local.

1. Problem Statement

Large-scale Power Quality (PQ) measurement campaigns generate vast amounts of multivariate data. Identifying systematic dependencies within this data is challenging due to:

Data Volume: The sheer number of possible combinations between parameters (e.g., harmonics, flicker) and measurement sites makes manual analysis impossible.
Temporal Variability: Correlations between PQ parameters are not constant; they fluctuate daily. Calculating a single correlation over a long period can lead to misinterpretation by masking these dynamics.
Visualization Complexity: Traditional correlation matrices become unreadable when analyzing multiple parameters across dozens of sites, obscuring robust patterns and recurring relationships.

2. Methodology

The authors propose a robust, automated framework adapted from existing long-term analysis methods to handle shorter observation periods and large datasets. The methodology consists of four main stages:

A. Data Preprocessing and Correlation Calculation

Dataset: The study utilizes data from 85 measurement sites in the German transmission system (110 kV, 220 kV, and 380 kV levels) over a 30-day period.
Parameters: 20 PQ parameters are analyzed, including Voltage/Current RMS, flicker, unbalance, Total Harmonic Distortion (THD), and odd-order harmonics (3rd to 15th).
Metric: The Spearman rank correlation coefficient is used instead of Pearson's. This non-parametric measure captures monotonic dependencies (consistent increases/decreases) rather than just linear relationships.
Daily Calculation: Correlations are calculated daily to capture temporal fluctuations, rather than assuming a static relationship over the entire period.

B. Robust Averaging (Fisher's z-Transformation)

To derive a single representative correlation coefficient from daily fluctuations and phase combinations:

Temporal Averaging: Daily coefficients ( $r_i$ ) are averaged using Fisher's z-transformation to account for the non-linear distribution of correlation coefficients:
$\bar{r} = \tanh \left( \frac{1}{N} \sum_{i=1}^{N} \text{arctanh}(r_i) \right)$
Phase Aggregation: The mean coefficients are further averaged across all phase combinations (e.g., L1-L1, L1-L2) to produce a single representative value per parameter pair per site.

C. Aggregation Across Sites

To identify consistent patterns across the network, the authors define an Aggregation Matrix ( $A$ ):

A significance threshold ( $\tau_r$ , e.g., 0.7) is applied.
The matrix counts how often a specific parameter pair exceeds this threshold across all $M$ measurement sites.
This reduces the complexity of analyzing 85 individual matrices into one compact matrix representing the consistency of correlations.

D. Visualization Techniques

To interpret the aggregated data, the correlation matrix is transformed into a distance matrix ( $D_{ij} = 1 - A_{ij}$ ) and visualized using:

Hierarchical Clustering (Dendrograms): Uses average-linkage to group parameters/sites based on correlation strength. Early merges indicate strong, frequent correlations.
Multidimensional Scaling (MDS): Projects the high-dimensional distance matrix into a 2D space, preserving pairwise distances to visualize clusters and outliers geometrically.

3. Key Contributions

Adaptation for Short-Term Data: Modifies existing long-term frameworks to effectively analyze shorter measurement periods by emphasizing daily correlation dynamics.
Statistical Rigor: Introduces a two-stage Fisher z-transformation process to robustly average correlations across time and phases, avoiding the pitfalls of simple arithmetic means.
Scalable Visualization: Demonstrates how aggregation matrices combined with MDS and clustering can distill complex, high-dimensional PQ data into interpretable visual structures.
Network-Wide Insight: Extends correlation analysis from intra-site (parameter-to-parameter) to inter-site (site-to-site) relationships, revealing spatial propagation patterns.

4. Key Results

The methodology was applied to the 85-site dataset, yielding the following insights:

A. Parameter-to-Parameter Correlations

Strong Consistency: Total Harmonic Distortion (THD) is strongly and consistently correlated with the 5th and 7th harmonics across more than 50% and 33% of sites, respectively.
Flicker and Current: Strong correlations exist between current quality parameters (THC) and specific harmonics (3rd, 9th) and flicker ( $U_{pst}$ ).
Isolation of High Orders: Higher-order harmonics (9th, 15th) showed rare or no systematic correlations, appearing as isolated nodes in the clustering dendrogram.

B. Site-to-Site Correlations

The analysis of correlations between the 85 sites revealed three distinct groups:

Group 1 (Isolated): Sites with minimal correlation to others (mostly 110 kV).
Group 2 (EHV Cluster): Sites at 220 kV and 380 kV levels showing moderate mutual correlation (20–60% of parameters).
Group 3 (HV Cluster): Sites at 110 kV showing the highest mutual correlation (40–75% of parameters).

C. Spatial Propagation

Voltage Harmonics & Flicker: The 3rd voltage harmonic ( $U_{03}$ ) and flicker ( $U_{pst}$ ) exhibited the highest spatial propagation, with significant correlations across 47% and 33% of all site pairs, respectively.
Conclusion: Low-order harmonics and flicker propagate widely across the transmission network, whereas higher-order harmonics tend to be localized phenomena.

5. Significance

Root Cause Analysis: The ability to distinguish between localized and propagated anomalies helps engineers pinpoint the source of PQ issues (e.g., a specific feeder vs. a system-wide event).
Measurement Optimization: By identifying highly correlated sites, utility operators can potentially reduce redundant measurement campaigns, optimizing resource allocation.
System Understanding: The visualization techniques provide a clear, geometric understanding of the power system's behavior, moving beyond raw data tables to actionable structural insights.
Future Applications: The framework lays the groundwork for "correlation-based segmentation" to identify operational regimes and temporal changes in grid behavior.