💻 computer science

Evaluating Hyperparameter Sensitivity in Temporal Convolutional Networks for Short-Term Electric Load Forecasting Using Grid Search

This study utilizes a systematic grid search on Queensland's half-hourly load data to demonstrate that optimizing five key hyperparameters in Temporal Convolutional Networks—specifically identifying a configuration with a 336-step input size and 0.0005 learning rate—can reduce forecasting error by over 55% compared to a reference model, highlighting that coordinated parameter tuning is more critical than simply increasing model capacity for short-term electric load prediction.

Original authors: Tuan Anh Nguyen, Thanh Ngoc Tran

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tuan Anh Nguyen, Thanh Ngoc Tran

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict exactly how much pizza your family will eat tomorrow. You know that people get hungry at different times, that weekends look different from weekdays, and that the weather might change their appetites. If you guess wrong, you might end up with a fridge full of wasted cheese or a very hungry family. This is the same challenge faced by power companies, but instead of pizza, they are guessing how much electricity people will use. Getting this wrong is a big deal: if they guess too low, the lights might flicker or go out; if they guess too high, they waste money and resources. To make these guesses, scientists use special computer programs called "neural networks." Think of these as digital brains that learn from the past to predict the future. One popular type of brain is called a Temporal Convolutional Network, or TCN. It's like a super-smart detective that looks at a long history of electricity usage to spot patterns. But here's the catch: just like a detective needs the right magnifying glass and the right amount of coffee to work best, these computer brains need their settings tweaked perfectly. If the settings are off, even a super-smart brain might make a silly mistake.

This paper is like a massive, organized experiment where two researchers, Tuan Anh Nguyen and Thanh Ngoc Tran, decided to play "what if" with these computer brains. They wanted to find the perfect recipe for the TCN to predict electricity usage for the next day in Queensland, Australia. They didn't just guess; they used a method called "Grid Search," which is like trying every single combination of ingredients in a cookbook to see which one makes the best cake. They tested 243 different versions of the computer brain, changing five specific knobs on the machine: how far back in history the brain looks (input size), how fast it learns (learning rate), how big its internal memory is (encoder and decoder sizes), and how many pieces of data it processes at once (batch size).

The results were surprising and very clear. The researchers found that the "best" setting wasn't about making the brain as huge and complex as possible. In fact, they discovered that simply making the brain bigger didn't always make it smarter. The most accurate prediction came from a specific combination: a brain that looked back 336 time steps (about 7 days of history), learned very slowly and carefully (a learning rate of 0.0005), had a compact memory for understanding patterns (64 hidden units), and a slightly larger memory for making the final guess (128 hidden units). This specific setup made a mistake of only 1.841%, which is incredibly accurate.

To put this in perspective, the researchers compared this "champion" setup to a "reference" setup that used more standard, middle-of-the-road settings. The standard setup made a mistake of 4.096%. By finding the right combination of settings, the researchers reduced the error by more than half—specifically, a 55.05% improvement. This proves that tuning the settings is more important than just building a bigger model.

The study also ruled out a few common assumptions. For instance, they found that making the "hidden size" (the brain's internal complexity) bigger didn't consistently make the predictions better. Sometimes a smaller, more focused brain worked better than a giant, cluttered one. They also found that the "batch size" (how many data chunks the computer processes at once) didn't actually change the accuracy in their specific test, because they were only looking at one single electricity grid. However, the "learning rate" was the most critical knob to turn; if they turned it too fast (0.005), the brain got confused and made worse guesses. If they turned it too slow, it was safe but not as sharp. The sweet spot was a slow, steady pace.

In the end, the paper suggests that the secret to predicting electricity isn't just having a super-computer; it's about knowing exactly how to tune it. The best results came from a balanced approach: looking at a full week of history, learning slowly, and using a compact but well-organized brain structure. While the researchers admit this was tested on just one specific dataset and one specific day, the lesson is clear: in the world of electricity forecasting, the right settings matter more than raw power.

Technical Summary: Evaluating Hyperparameter Sensitivity in Temporal Convolutional Networks for Short-Term Electric Load Forecasting Using Grid Search

Problem Statement
Short-term electric load forecasting is critical for the secure and economical operation of modern power systems, influencing generation scheduling, reserve allocation, and renewable energy coordination. While Temporal Convolutional Networks (TCNs) have emerged as effective deep learning architectures for capturing sequential patterns due to their causal convolutions, dilated receptive fields, and parallel computation capabilities, their performance is highly sensitive to hyperparameter configurations. Previous studies often focus on proposing new architectures or identifying a single "best" configuration via optimization methods, providing limited insight into the individual and interaction effects of specific hyperparameters. This lack of systematic sensitivity analysis makes it difficult to reproduce results or understand why certain configurations succeed or fail across different datasets and settings.

Methodology
The study establishes a controlled, reproducible framework to investigate the sensitivity of five key TCN hyperparameters:

Input Size ( $L$ ): The length of the historical observation window (candidates: 96, 144, 336 steps, corresponding to 2, 3, and 7 days).
Learning Rate ( $\eta$ ): The optimizer step size (candidates: 0.0005, 0.001, 0.005).
Encoder Hidden Size ( $C_e$ ): The representational capacity of the temporal convolutional encoder (candidates: 64, 128, 256).
Decoder Hidden Size ( $C_d$ ): The width of the MLP decoder (candidates: 64, 128, 256).
Batch Size ( $B$ ): The series-level batch parameter in the NeuralForecast data loader (candidates: 16, 32, 64).

A full factorial Grid Search was conducted, evaluating all $3^5 = 243$ unique configurations. The experiments utilized half-hourly electricity demand data from Queensland, Australia, covering the period from January 2015 to December 2021. The dataset was split chronologically into a 28-day training set (1,344 observations) and a 1-day holdout test set (48 observations). The models were implemented using the NeuralForecast library (v3.1.9) with a PyTorch backend. All trials used a fixed random seed (42), a 48-step forecasting horizon (one-day-ahead), and Mean Squared Error (MSE) as the training loss. Performance was evaluated using Mean Absolute Percentage Error (MAPE) on the holdout set.

Key Contributions

Controlled Sensitivity Analysis: The study provides a full factorial evaluation of 243 configurations under identical experimental conditions, isolating the effects of specific hyperparameters on TCN performance.
Marginal and Interaction Effects: Unlike studies that report only the optimal configuration, this work analyzes the marginal influence of each parameter and their pairwise interactions (e.g., how input size interacts with learning rate).
Reproducible Framework: The authors detail a deterministic workflow using the NeuralForecast library, ensuring that performance variations are attributable to hyperparameter settings rather than data partitioning or stochastic training variations.
Practical Configuration Guidance: The study offers evidence-based recommendations for configuring TCNs in short-term load forecasting, challenging the assumption that larger model capacity always yields better accuracy.

Results
The 243 configurations produced MAPE values ranging from 1.841% to 5.455%, with an average of 3.499%.

Best Configuration: The optimal setup achieved a MAPE of 1.841% with the following parameters: Input Size = 336 (7 days), Learning Rate = 0.0005, Encoder Hidden Size = 64, Decoder Hidden Size = 128, and Batch Size = 16.
Comparison to Reference: Compared to a reference configuration (Input=144, LR=0.001, Encoder=128, Decoder=128, Batch=32) which yielded a MAPE of 4.096%, the best configuration reduced the error by 55.05%.
Hyperparameter Sensitivity:
- Learning Rate: Exhibited the clearest marginal effect. The lowest rate (0.0005) consistently produced the lowest average error, while the highest rate (0.005) generally degraded performance.
- Input Size: A 7-day window (336 steps) yielded the lowest marginal mean error but showed the highest variance, indicating that longer historical context is beneficial only when paired with appropriate optimization settings.
- Hidden Dimensions: Increasing encoder or decoder size did not consistently improve accuracy. The best-performing model used a compact encoder (64) and a moderate decoder (128). Larger dimensions (e.g., 256) did not guarantee better results and sometimes increased error.
- Batch Size: In this single-series experiment, batch size had no marginal effect on accuracy, as the parameter controls series-level batching rather than temporal window sampling.
Computational Cost: There was no strong correlation between total execution time and forecasting accuracy. The best configuration required only a modest increase in runtime compared to the fastest trial.
Forecasting Behavior: The best model accurately captured the overall daily load profile ( $R^2 = 0.9528$ ) but exhibited smoothing effects during rapid load transitions (morning and afternoon ramps), leading to localized errors.

Significance and Claims
The paper claims that effective TCN performance in short-term load forecasting depends on coordinated hyperparameter settings rather than simply increasing model capacity. The study demonstrates that:

Inappropriate hyperparameter selection can lead to substantial degradation in forecasting accuracy (up to a 55% difference in error rates).
Larger hidden dimensions do not necessarily reduce prediction errors; a compact encoder with a moderate decoder can outperform larger architectures.
The learning rate is a critical driver of stability and accuracy, particularly when utilizing longer historical input windows.
Systematic sensitivity analysis is necessary to understand TCN behavior, as reporting only the best configuration obscures the complex interactions between parameters.

The authors conclude that while the findings provide practical guidance for configuring TCN-based models using the NeuralForecast framework, the results are specific to the single regional dataset, fixed training period, and experimental protocol used. Future work should explore these sensitivities across multiple regions, seasons, and with the inclusion of exogenous variables.

More like this