Testing the Efficacy of Hyperparameter Optimization Algorithms in Short-Term Load Forecasting

This study evaluates the performance and runtime efficiency of five hyperparameter optimization algorithms (Random Search, CMA-ES, Bayesian Optimization, PSO, and NGOpt) in tuning XGBoost for short-term load forecasting using the Panama Electricity dataset, revealing significant runtime advantages over Random Search while noting that Bayesian Optimization yielded the lowest accuracy in univariate models.

The Gist

Imagine you are trying to predict exactly how much electricity a country will need next hour. This is like trying to guess how many people will show up to a party so you don't run out of snacks or have too much left over. If you guess wrong, the power grid could get stressed (like a party getting too crowded) or energy gets wasted (like throwing away food).

This paper is about finding the best way to tune the "knobs and dials" on a computer program (specifically one called XGBoost) to make these predictions as accurate as possible.

Here is the breakdown of the study using simple analogies:

1. The Problem: Tuning a Radio

Think of the XGBoost program like a high-tech radio. To get the clearest signal (the most accurate prediction), you have to turn the knobs (called hyperparameters) just right.

• The Challenge: There are thousands of possible positions for these knobs. If you turn them randomly, you might get lucky, but it usually takes forever.
• The Goal: The researchers wanted to find the fastest and smartest way to find the "perfect station" without wasting time.

2. The Contestants: Five Different Search Strategies

The researchers tested five different "search strategies" (algorithms) to see which one could tune the radio best. Imagine these as five different people looking for a lost key in a giant field:

1. Random Search: This person closes their eyes and picks a spot in the field to dig randomly. They might find the key, but they might also dig in the same spot twice or miss the whole field. It's simple but inefficient.
2. CMA-ES (The Evolutionary Gardener): This person acts like a gardener. They plant seeds (guesses) in a few spots. If a seed grows well, they plant more seeds nearby. Over time, the "garden" evolves to focus only on the best spots.
3. Bayesian Optimization (The Smart Detective): This person builds a map based on what they've already found. If they find a clue that says "the key is likely in the north," they stop looking south. They use logic and past clues to narrow down the search.
4. PSO (The Swarm of Birds): Imagine a flock of birds looking for food. Each bird flies around, but they all talk to each other. If one bird finds a tasty worm, the whole flock adjusts its flight path to swarm that area. They share information to find the best spot quickly.
5. NGOpt (The Chameleon): This is a new, smart tool that can change its shape. It looks at the problem and decides, "Okay, today I'll act like the Detective; tomorrow I'll act like the Bird Swarm." It adapts to whatever the situation needs.

3. The Experiment: Two Types of Parties

The researchers tested these searchers in two different scenarios using real data from Panama:

• Univariate (The Solo Act): The searchers only looked at one thing: the history of electricity usage. It's like trying to guess the party size just by looking at how many people showed up last year.
• Multivariate (The Full Picture): The searchers looked at everything: electricity history, plus the weather (is it hot? is it raining?), holidays, and school days. It's like guessing the party size by looking at the weather, the calendar, and last year's attendance.

4. The Results: Who Won?

The researchers measured two things: Accuracy (how close the guess was) and Speed (how long it took to find the answer).

• Speed is King: The "Smart" searchers (CMA-ES, Bayesian, PSO, NGOpt) were drastically faster than the "Random" searcher. Random Search was like a turtle; the others were like race cars.
• The Univariate Surprise: When looking only at electricity history (no weather data), the Smart Detective (Bayesian) actually did the worst job at accuracy. It seems like without extra clues, the Detective got confused.
• The Multivariate Win: When they added weather and holiday data, everyone got better. The Smart Detective finally found its footing and performed just as well as the others.
• The New Kid: The Chameleon (NGOpt) did very well, proving that an adaptable tool is a powerful thing to have.

5. The Big Takeaway

The main lesson from this paper is: Don't just guess randomly.

If you are trying to predict energy needs (or anything else in the future), using a "smart" method to tune your computer model is much better than just throwing darts at a board.

• Smart methods save time: They find the best settings much faster.
• Context matters: Smart methods work best when they have extra information (like weather) to help them make decisions.
• One size doesn't fit all: Sometimes a specific method (like the Detective) needs more data to work well, while others (like the Birds) are more consistent.

In short: To predict the future accurately, you need the right tools, and the "smart" tools that learn from their mistakes are much faster and more reliable than just guessing.

Technical Summary

1. Problem Definition

The study addresses the challenge of Short-Term Load Forecasting (STLF), which involves predicting electricity demand at hourly intervals (from minutes to days ahead). Accurate STLF is critical for grid stability, resource allocation, and energy efficiency.

• Core Challenge: While machine learning models like XGBoost are effective for STLF, their performance is heavily dependent on the configuration of hyperparameters (e.g., learning rate, tree depth).
• Gap in Literature: Although various Hyperparameter Optimization (HPO) algorithms exist, there is a lack of comprehensive, statistically rigorous comparisons of these algorithms across different sample sizes and data configurations (univariate vs. multivariate) specifically within the STLF domain.

2. Methodology

A. Dataset

• Source: Panama National Electricity Demand dataset.
• Scope: Hourly measurements from January 2015 to June 2020 ($n=48,049$).
• Features:
  • Target: National electricity load (MWh).
  • Continuous Variables: Temperature, relative humidity, wind speed, precipitation.
  • Categorical Variables: Holidays and school days.
• Preprocessing: Missing values were absent; duplicate indices were removed; features were normalized using Min-Max scaling.

B. Forecasting Model (Surrogate)

• Algorithm: XGBoost (Extreme Gradient Boosting).
• Rationale: Selected for its proven efficacy in time series tasks, sensitivity to hyperparameter tuning, and relatively short runtime compared to deep learning models (e.g., LSTMs), making it ideal for evaluating HPO efficiency.
• Configurations:
  • Univariate: Predicts $Y(t+1)$ using only historical demand $Y(t-S):t$.
  • Multivariate: Predicts $Y(t+1)$ using historical demand plus weather and calendar features $X(t-S):t$.

C. Hyperparameter Optimization (HPO) Algorithms

Five distinct HPO strategies were compared:

1. Random Search: The baseline "uninformed" method.
2. CMA-ES (Covariance Matrix Adaptation Evolution Strategy): An evolutionary algorithm adapting a multivariate normal distribution.
3. Bayesian Optimization: A sequential model-based approach using Gaussian Processes to balance exploration and exploitation.
4. PSO (Particle Swarm Optimization): Simulates social behavior (swarm intelligence) to find optimal solutions.
5. NGOpt (Nevergrad Optimizer): A meta-algorithm that dynamically selects and combines optimization strategies based on the problem landscape.

D. Experimental Setup

• Platform: Google Colab Pro+ (A100 GPU).
• Search Space: 50 experiments per configuration with early stopping (set to 20 iterations) for all except Random Search (which used 5-fold cross-validation).
• Optimization Objective: Minimize RMSE.
• Evaluation Metrics:
  • Accuracy: Mean Absolute Percentage Error (MAPE) and $R^2$.
  • Efficiency: Total runtime (time to find optimal hyperparameters).
• Statistical Analysis: A non-parametric Kruskal-Wallis test followed by pairwise comparisons with Bonferroni correction ($\alpha = 0.05$) to determine statistical significance of performance differences.

3. Key Contributions

1. Systematic Evaluation: A comparative analysis of five state-of-the-art HPO algorithms (including the novel application of NGOpt for XGBoost in STLF) across both univariate and multivariate settings.
2. Scalability Visualization: Performance plots were generated to visualize how MAPE, $R^2$, and runtime evolve as sample sizes increase from 1,000 to 20,000 observations.
3. Statistical Rigor: The study employs the Kruskal-Wallis test to move beyond simple metric comparison, providing statistically validated conclusions regarding the superiority of specific algorithms.

4. Results

A. Runtime Efficiency

• Significant Advantage: All intelligent HPO algorithms (CMA-ES, Bayesian, PSO, NGOpt) demonstrated significantly shorter runtimes compared to Random Search.
• Fastest Algorithms: CMA-ES and Bayesian Optimization were generally the fastest, while Random Search was the slowest due to its lack of guided search and early stopping mechanisms.
• Statistical Significance: The Kruskal-Wallis test confirmed these runtime differences were statistically significant ($p < 0.05$).

B. Forecasting Accuracy

• Univariate Models:
  • Bayesian Optimization surprisingly exhibited the lowest accuracy (lowest $R^2$) among the tested methods in univariate settings.
  • Performance differences between other algorithms were often not statistically significant, except for $R^2$ where Bayesian underperformed compared to NGOpt.
• Multivariate Models:
  • All HPO algorithms improved XGBoost performance as sample sizes increased (decreasing MAPE trend).
  • Convergence: In multivariate settings, the statistical differences in accuracy between HPO algorithms became insignificant. This suggests that when rich contextual features (weather, calendar) are available, even simpler optimization strategies can leverage the data effectively, and Bayesian optimization's advantage diminishes.

C. Summary of Metrics (Table IV Insights)

• MAPE: Random Search and CMA-ES showed similar mean MAPE in univariate (0.115), while multivariate MAPE dropped significantly (0.092) across all methods.
• $R^2$: Bayesian Optimization showed high variance and negative $R^2$ values in some univariate instances, indicating poor fit compared to Random Search ($R^2 \approx 0.29$).
• Runtime: Random Search took ~93 minutes (univariate) vs. ~4–7 minutes for intelligent methods.

5. Significance and Conclusion

• Practical Implication: For STLF tasks, intelligent HPO algorithms are superior to Random Search primarily in terms of computational efficiency, reducing the time required to tune models by an order of magnitude.
• Contextual Insight: The efficacy of specific HPO algorithms depends on the data context. Bayesian Optimization, often touted as the gold standard, may underperform in univariate STLF tasks but becomes competitive when multivariate features are introduced.
• Limitations: The study was limited to a single forecasting algorithm (XGBoost) and a single dataset due to computational constraints. It did not explore other hyperparameters (e.g., regularization terms) or other HPO strategies.
• Future Work: The authors suggest future research should focus on interpretability, cross-regional comparisons, and testing HPO scalability across different time granularities (daily/weekly) and building types.

Final Verdict: The study confirms that while Random Search is a valid baseline, CMA-ES, PSO, and NGOpt offer a robust balance of speed and accuracy for STLF. However, the choice of HPO algorithm should be guided by the availability of external features (multivariate vs. univariate) rather than assuming a single algorithm is universally superior.