Bridging the climate to energy data gap: simulated annealing for representative climate year selection

This study proposes and validates a simulated annealing optimization method, utilizing the seasonal sliced Wasserstein distance, to select highly representative subsets of climate years from large ensembles, significantly outperforming current practices and alternative algorithms to provide robust, unbiased inputs for energy system modeling.

The Gist

Imagine you are trying to design a power grid that can handle the weather for the next 30 years. The problem is that the weather is chaotic and unpredictable. Climate scientists have run supercomputer simulations that generated 180 different possible years of weather data to show every possible scenario (from super windy years to droughts).

However, the computer models used to design the actual power grid are very heavy and slow. They can't process 180 years of data at once; they can only handle a tiny handful, maybe 5 or 30 years.

The big question is: Which specific years should we pick?

If you pick the wrong years, you might build a grid that works great in a mild summer but collapses during a cold, windless winter. If you pick the wrong years, you could waste billions of dollars on the wrong infrastructure.

The Problem with Current Methods

Right now, many energy planners pick years somewhat randomly or by just looking at the "average" year. The authors of this paper say this is like trying to understand a whole library by reading just one random page. It often misses the extreme events (like a "Dunkelflaute"—a period of no wind and no sun) that are crucial for planning.

The Solution: A "Smart Search" (Simulated Annealing)

The authors propose a new method called Simulated Annealing.

The Analogy:
Imagine you are in a vast, foggy mountain range, and you want to find the absolute lowest valley (the best set of years).

• Random Search is like throwing a dart at a map and walking there. You might get lucky, but you'll probably miss the deepest valley.
• K-Medoids (the old standard) is like grouping the mountains into clusters and picking the center of each group. It's okay, but it might miss the specific shape of the terrain.
• Simulated Annealing is like a hiker who is smart but also willing to take a risk.
  • The hiker starts at a random spot.
  • They look around. If they find a lower spot, they move there.
  • Crucially: Sometimes, they might take a step uphill (a worse spot) just to see if there is an even deeper valley on the other side of that hill.
  • As the "hike" goes on, they get less willing to take those risky uphill steps and start focusing on finding the absolute bottom.
  • This prevents them from getting stuck in a small, shallow dip (a local minimum) and missing the true lowest point (the global minimum).

How They Measure "Goodness"

How do they know if their chosen 5 or 30 years are actually good? They use a mathematical tool called the Seasonal Sliced Wasserstein Distance.

The Analogy:
Think of the 180 years of weather data as a giant, complex smoothie made of many ingredients (wind, sun, temperature, electricity demand).

• A simple average might just check if the total amount of strawberries is right.
• This new tool checks:
  1. The Ingredients: Is the right amount of wind and sun there?
  2. The Mix: Do the ingredients blend correctly? (e.g., Does high wind usually happen with low sun? Or do they happen together?)
  3. The Timing: Is the mix right for winter and summer separately? (A windy summer is great, but a windy winter is even better for heating. If you pick years that are windy in summer but calm in winter, you fail the test).

The tool calculates a "score" of how different your small smoothie (the selected years) is from the giant smoothie (all 180 years). The lower the score, the better the match.

What They Found

The researchers tested their "Smart Search" method against random guessing, filtered guessing, and the old clustering method across three scenarios:

1. Just the Netherlands (30 years).
2. All of Europe (30 years).
3. All of Europe (5 years).

The Results:

• The Winner: The "Smart Search" (Simulated Annealing) consistently found the best sets of years.
• The Magic Multiplier: When they picked just 30 years using this method, those 30 years were so representative that they acted like 130 to 140 years of data. They got 4 to 5 times more "value" out of the data than they physically had.
• Better than Current Practice: The method they used is 2.5 to 3.5 times better than the current standard used by major European energy organizations (ENTSO-E).
• Consistency: Unlike other methods that rely heavily on "luck" (getting a good result just by chance), this method works reliably every time you run it.

The Bottom Line

This paper doesn't just say "pick better years." It provides a specific, mathematically proven recipe (Simulated Annealing + a specific scoring tool) to ensure that when energy companies build the grid for the future, they aren't gambling on a lucky guess. They are using a tiny, carefully selected sample that perfectly mirrors the complex, chaotic reality of the full climate.

One final note on the "Year": The paper also suggests defining a "year" from April 1st to March 31st (instead of January to December). Why? Because this keeps the winter together in one block. Since winter is the most stressful time for the power grid (heating + less sun), splitting winter across two calendar years would break the data and make it harder to plan for those critical cold snaps.

Technical Summary

Technical Summary: Bridging the Climate to Energy Data Gap via Simulated Annealing

Problem Statement
Energy system models are increasingly reliant on accurate climate inputs to plan for weather-dependent renewable sources. However, a fundamental mismatch exists between climate science and energy system modeling. Climate science utilizes large ensembles (hundreds to thousands of simulated years) to capture the full distribution of possible outcomes, including extremes. Conversely, computationally demanding power system models (handling unit commitment, network constraints, and market dynamics at hourly resolution) can typically process only a handful of years (1–30).

Current practices for selecting these representative years, such as those used by ENTSO-E in the European Resource Adequacy Assessment (ERAA), often rely on unvalidated methods like random selection, pseudo-random selection around a target period, or K-Medoids clustering. These approaches lack explicit representativeness criteria, risking biased investment decisions and leaving energy systems unprepared for plausible weather conditions that are excluded by poor methodology. The selection of a small subset from a large ensemble is a combinatorial problem with a "combinatorial explosion" of possibilities (e.g., selecting 30 years from 180 yields $\approx 1.3 \times 10^{34}$ combinations), rendering exhaustive search infeasible.

Methodology
This study proposes Simulated Annealing (SA) as an optimization method to select representative subsets of complete climate years from large ensembles. The approach is evaluated against three alternative methods: Random Search (RS), Filtered Random Search (FRS), and K-Medoids clustering.

• Data Source: The study utilizes the Pan-European Climate Database (PECD), specifically 180 climate years (6 CMIP6 models $\times$ 30 years under the SSP2-4.5 scenario, 2016–2045).
• Variables: Selection is performed on derived energy-impact variables rather than raw meteorological fields to account for non-linearities. These include offshore/onshore wind production, solar production, various hydropower types, electricity demand, and residual load, aggregated to daily means at the national (NUTS0) level.
• Cost Function (Representativeness Metric): The study employs the Seasonal Sliced Wasserstein Distance (SSWD).
  • Wasserstein Distance: An optimal transport metric quantifying the "work" to transform one probability distribution into another, capturing joint dependence structures and correlations between variables.
  • Sliced Approximation: To handle high-dimensional data efficiently, the multivariate data is projected onto random 1D lines, and the 1D Wasserstein distance is computed and averaged.
  • Seasonal Extension: To prevent seasonal biases (e.g., a windy summer compensating for a low-wind winter), the dataset is split into extended winter (Oct–Mar) and summer (Apr–Sep). The SSWD is defined as the maximum of the two seasonal distances, ensuring both seasons are adequately represented.
• Algorithms:
  • Simulated Annealing: A stochastic optimization algorithm that iteratively swaps years in a candidate subset, accepting worse solutions with a decreasing probability to escape local minima.
  • K-Medoids: Clusters years and selects the most central member (medoid) of each cluster.
  • Filtered Random Search: Applies cheap pre-filters (mean difference and energy distance) to reject poor candidates before computing the expensive SSWD.
• Experimental Design: Three case studies were conducted: (1) 30 years for the Netherlands, (2) 30 years for Europe, and (3) 5 years for Europe. Each method was run 25 times to assess consistency using "luck" and "fragility" factors.

Key Results
Across all three case studies, Simulated Annealing consistently produced the most representative subsets:

• Performance: SA achieved the lowest SSWD scores in all cases. In the 30-year European case, the SA subset was roughly 2.5–3.5 times more representative than the current ENTSO-E practice (ERAA25), despite the latter using 36 years compared to SA's 30.
• Effective Sample Size: The SA-selected subsets achieved an "effective sample size" four to five times their actual size. For instance, the best 30-year SA subset for the Netherlands was statistically equivalent to a median-quality random draw of 140 years.
• Consistency: SA demonstrated low "luck" and "fragility" factors, indicating stable performance regardless of initial conditions. In contrast, Filtered Random Search performed well in single-draw scenarios but degraded significantly in high-dimensional (European) cases, often performing worse than plain random search due to over-restrictive pre-filtering.
• Comparison: K-Medoids consistently ranked second but did not directly optimize for the joint distribution representativeness in the same way SA did.

Significance and Claims
The paper claims that the proposed SA+SSWD approach bridges the gap between climate science (large ensembles) and energy system modeling (small subsets) by providing a validated, application-agnostic method for climate year selection.

• Objective Quantification: The study argues that representativeness must be objectively quantified and optimized rather than relying on convention or unvalidated random draws.
• Robustness: By optimizing the input dataset before energy-specific processing (a bottom-up approach), the method ensures that the source data entering any energy study is validated for climate variability. This prevents downstream models from being biased by unrepresentative starting points.
• General Applicability: The methodology is not limited to energy system modeling; it is applicable to any domain requiring the selection of representative complete periods from large datasets, including wind yield assessments, building energy simulations, hydrological studies, and agricultural impact modeling.
• Practical Recommendations: The authors recommend using Simulated Annealing with the SSWD metric, running multiple iterations to ensure near-optimality, and defining climate years as hydrological years (April 1–March 31) to preserve winter resource adequacy stress events.

The study concludes that while SA does not guarantee a global optimum (due to the infeasibility of brute-force evaluation), its consistent performance across multiple runs and its ability to significantly outperform current industry practices make it a superior standard for selecting representative climate years.