⚡ electrical engineering

Synthetic weather ensembles reveal hidden risks of capacity shortfalls in bulk power systems

This paper introduces a universal, open-source synthetic weather data augmentation technique that generates unlimited meteorological ensembles to overcome the statistical biases of historical records in non-stationary climate systems, revealing significant hidden capacity shortfalls and extreme price risks in U.S. bulk power systems that traditional models fail to detect.

Original authors: Jordan Kern, Duc-Huy Pham, Jingwei Qian, Troy Wibowo, Dimitrios Floros, Dalia Patiño-Echeverri

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jordan Kern, Duc-Huy Pham, Jingwei Qian, Troy Wibowo, Dimitrios Floros, Dalia Patiño-Echeverri

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to plan a massive, city-wide party where the weather is the most important guest. You need to know exactly how many heaters to bring if it freezes, or how many fans if it swelters, but you can't predict the future. This is the daily challenge for the people who run our power grids. They are the "party planners" of the electricity world, tasked with making sure there is enough power for everyone, from the moment you wake up to the second you go to sleep. To do this, they usually look at the past, studying old weather records like a detective looking at a case file. They ask, "What was the hottest day in history? How cold did it get?" and then they build enough power plants to handle those worst-case scenarios.

However, there is a problem with only looking at the past. The climate is changing, and the "rules" of the weather are shifting. Relying solely on old records is like trying to predict the next big wave in the ocean by only looking at the ripples from yesterday; you might miss the massive tsunami that's coming. Scientists call this "non-stationarity," which is a fancy way of saying the past doesn't perfectly predict the future anymore. If planners guess wrong, they might build too few power plants, leading to blackouts when people need heat or air conditioning the most. This is a high-stakes game where the cost of being wrong is measured in billions of dollars and the comfort (or safety) of millions of people.

Enter a team of researchers who decided to stop just looking at the past and start inventing the future. In their new study, they introduced a clever digital tool that acts like a "weather time machine" or a "what-if generator." Instead of waiting for nature to provide a new, extreme weather event to test their power grid, they used math to create thousands of new weather scenarios that have never happened before but are still physically possible. They call this "synthetic weather augmentation." Think of it like a video game designer who knows the rules of physics; they can generate a level with a storm so intense that no player has ever seen it, just to see if the game characters can survive it.

The researchers used this tool to stress-test the massive power grid that covers the Eastern United States. They fed their computer a small slice of recent weather data (from 2004 to 2022) and asked it to imagine 500 years of future weather based on those patterns. The result was a shock. The "synthetic" weather they generated included heatwaves and cold snaps that were significantly more extreme than anything recorded in the last 80 years. When they ran their power grid simulations with these extreme scenarios, the system started to crack. They found that in three major regions—PJM, MISO, and NYIS—there were "hidden" gaps in power capacity. Specifically, the grid was short by at least 2.1 gigawatts in PJM, 4.0 gigawatts in NYIS, and a whopping 6.6 gigawatts in MISO.

To put those numbers in perspective, the researchers calculated that fixing these hidden shortages would require billions of dollars in new investments. For example, MISO alone might need to build enough new power plants to cost about $6.5 billion. Without this new tool, planners might have looked at the old records, shrugged, and said, "We have enough power," only to be caught off guard when a real, record-breaking heatwave hits. The study suggests that by using these synthetic weather ensembles, we can uncover these invisible risks before they become blackouts. It's not a guarantee that these extreme events will happen tomorrow, but the simulations show that relying only on history is a dangerous gamble. By creating a library of "what-if" storms, we can build a grid that is ready for the unexpected, ensuring that when the real extreme weather arrives, the lights stay on.

Technical Summary: Synthetic Weather Ensembles Reveal Hidden Risks of Capacity Shortfalls in Bulk Power Systems

Problem Statement
Bulk power system planning relies on characterizing weather-based uncertainties in electricity supply and demand to ensure resource adequacy. Traditionally, planners subject models to past observations (reanalysis) or forward-looking global climate model outputs. However, relying on fixed historical datasets presents a critical limitation: in non-stationary systems like Earth's climate, increasing sample sizes by simply extending historical records can introduce statistical bias. For instance, using data from 1941–2020 to plan for a 2050 climate state may misrepresent the likelihood of extreme heat or cold events. Conversely, using narrow future climate windows risks missing system-defining events. Furthermore, empirical datasets are often "blind" to specific compound extreme events (simultaneous extremes across regions) that pose significant risks to interconnected continental-scale power systems. Existing AI-based synthetic weather generators (e.g., SEEDS, GenCast) often lack public availability or are constrained to short-to-medium range forecasts and specific historical reanalysis datasets (ERA5), limiting their utility for long-term resource adequacy planning.

Methodology
The authors propose a universal, open-source framework for synthetic weather data augmentation designed to expand underlying empirical records by orders of magnitude while preserving their statistical and temporal characteristics. The framework operates through seven core components:

Dimensionality Reduction: Gridded weather data (temperature, dew point, wind speed, solar irradiance losses) are aggregated into weighted averages for specific balancing authorities. Weights are determined by population centers (for load) or installed capacity (for renewables) to mitigate multicollinearity and "n < p" issues common in predictive modeling.
Time Series Decomposition: Daily data are decomposed into a deterministic expected value (seasonal cycle) and stochastic anomalies (residuals). The framework assumes the expected value remains constant for a chosen climate state, while anomalies drive the stochasticity.
Transformation: Anomalies are shifted to ensure positivity, then transformed via the Box-Cox method and normalized to a standard Gaussian distribution $N(0,1)$ on a monthly basis.
VAR Model Fitting: A Vector Autoregressive (VAR) model is fitted to the standardized multivariate anomalies. This captures cross-correlations among variables and locations, as well as autocorrelation, using coefficients determined by the Akaike Information Criteria (AIC).
Stochastic Simulation: The VAR model generates a large ensemble of synthetic multivariate noise. A stability mechanism is employed during the inverse transformation to prevent mathematical domain violations (e.g., ensuring $\lambda \tilde{s}_t + 1 > 0$ ) by searching within a 95% confidence interval for valid parameters if the optimal estimate fails.
Reconstruction: Synthetic anomalies are back-transformed to their original distributions and added to the deterministic seasonal signals to create new daily synthetic values.
Temporal Downscaling: Daily synthetic values are converted to hourly resolution by identifying empirical days with similar values and applying bias corrections, followed by a smoothing filter to ensure realistic transitions between days.

The resulting synthetic weather ensembles are fed into predictive machine learning models (trained on 2019–2022 data) to generate time series for total load, wind capacity factors, and solar capacity factors. These inputs drive a Direct Current Optimal Power Flow (DCOPF) model to simulate hourly operations and resource adequacy for the U.S. Eastern Interconnection.

Key Results
The framework was validated across four spatial scales (single balancing authority, state, interconnection, and contiguous U.S.) using two datasets: historical ERA5 (2004–2022) and future climate projections (Sup3rCC, 2022–2040).

Statistical Fidelity: The synthetic data closely replicated the statistical moments and probability distributions of the training data, evidenced by low Kolmogorov–Smirnov (KS) distances (mostly < 0.1) and minimal correlation errors (centered near zero) across variables and scales.
Tail Risk Expansion: Crucially, the synthetic ensembles generated extreme values beyond the bounds of the empirical training sets. Synthetic maximum net loads exceeded recent historical records (2004–2022) by an average of 28.5% and full historical records (1941–2022) by 21.5%.
Capacity Shortfall Identification: Analysis of three major balancing authorities (PJM, NYIS, MISO) revealed "hidden" capacity shortfalls not detectable via recent historical records alone:
- PJM: 2.1 GW shortfall (approx. $2 billion in capital investment).
- NYIS: 4.0 GW shortfall (approx. $3.9 billion).
- MISO: 6.6 GW shortfall (approx. $6.5 billion).
- These shortfalls imply potential impacts on 1.7, 3.3, and 5.5 million households, respectively, during extreme events.
Market Price Extremes: Simulated Locational Marginal Prices (LMPs) using synthetic data exhibited significantly wider ranges and higher maximums (e.g., reaching ~$45.56/MWh in NYIS) compared to empirical records, highlighting plausible high-cost operating conditions that historical data underestimates.

Significance and Claims
The paper claims that relying solely on historical datasets is fundamentally insufficient for quantifying system needs in a non-stationary climate. By leveraging large sample sizes through synthetic augmentation, the framework provides a practical means to stress-test power systems against rare but plausible compound extreme events. The authors assert that this approach offers an interpretable, open-source foundation for incorporating weather uncertainty into long-term planning. It enables planners to identify critical vulnerabilities and "missing" capacity that would otherwise remain obscured, thereby supporting more robust system design and preventing billions of dollars in under-investment that could lead to rolling blackouts during peak stress periods.

Limitations
The authors acknowledge several limitations: the model does not explicitly simulate the physical impacts of weather on transmission and generation capacity (e.g., line sag or thermal derating); it excludes hydrological variables crucial for hydropower and thermal cooling; and it generates ensembles representing a "typical year" rather than specific year-to-year climate cycle projections. Additionally, the stability mechanism used to prevent mathematical errors may inadvertently constrain the maximum allowable variance in some cases, requiring modelers to apply outlier detection or physical thresholds.

Technical Summary: Synthetic Weather Ensembles Reveal Hidden Risks of Capacity Shortfalls in Bulk Power Systems

More like this