Stealing Accuracy: Predicting Day-ahead Electricity Prices with Temporal Hierarchy Forecasting (THieF)

Imagine you are trying to predict the weather for the next day. You could look at the temperature for every single hour (8:00 AM, 9:00 AM, etc.), or you could look at the average temperature for the whole morning, the whole afternoon, or the entire day.

Usually, forecasters pick one of these views and stick to it. But what if the "hourly" view says it will be sunny at noon, while the "daily" view says it will be cloudy all day? If you make decisions based on just one of these conflicting stories, you might get wet or miss a great picnic.

This paper is about a clever new way to fix that confusion for electricity prices.

The Problem: The "Blind Spot"

Electricity markets are changing. In the past, people mostly traded electricity for a whole day. Now, they trade it for specific blocks of time (like 4-hour chunks or 12-hour chunks) and even individual hours.

Forecasters build models to predict these prices.

Model A might be great at guessing the price for 2:00 PM but terrible at guessing the average for the whole afternoon.
Model B might be great at the daily average but miss the tiny spikes at 2:00 PM.

If you just use Model A or Model B, you might make bad financial bets. The paper argues that these models are often "blind" to the bigger picture or the tiny details because they are looking at the data through a single lens.

The Solution: THieF (Temporal Hierarchy Forecasting)

The authors introduce a method called THieF (Temporal Hierarchy Forecasting). Think of it as a team of detectives working together to solve a mystery.

The Detectives (The Models): They build several different "detectives" (models) to look at the electricity prices. Some look at the prices hour-by-hour. Others look at 4-hour blocks. Others look at the whole day.
The Conflicting Reports: Each detective writes a report. Detective 1 says, "The price at 2 PM will be high!" Detective 2 says, "The average price for the 2 PM–6 PM block will be low!"
The Chief Detective (Reconciliation): This is the magic step. A "Chief Detective" (the reconciliation algorithm) takes all these conflicting reports and forces them to agree. It doesn't just pick one; it blends them mathematically. It says, "Okay, Detective 1, you were right about the spike, but Detective 2, you were right about the average. Let's adjust your numbers so they fit together perfectly."

The "Magic" Analogy: The Puzzle

Imagine you are trying to assemble a giant 1,000-piece puzzle of a landscape.

The Old Way: You try to assemble the whole thing at once, or maybe just the top half. If you miss a piece in the top half, the bottom half looks weird, and the whole picture is blurry.
The THieF Way: You assemble the puzzle in layers. You build the top half, the middle half, and the bottom half separately. Then, you lay them all on the table and smooth out the edges where they don't quite match. Suddenly, the picture becomes crystal clear. You recover details you would have missed if you only looked at one layer.

What Did They Find?

The researchers tested this on electricity markets in Germany and Spain over a very chaotic 4-year period (including the pandemic and the energy crisis). They used four different types of "detectives":

The Classicist: A simple, old-school math model (Linear Regression).
The Learner: A basic AI that learns patterns (Neural Network).
The Decision Tree: A model that asks "Yes/No" questions to make predictions (Gradient Boosting).
The Super-Brain: A massive, modern AI (Transformer) that usually needs huge amounts of data to learn.

The Results were amazing:

Every single detective got smarter when they used the "Chief Detective" (THieF) to reconcile their reports.
The accuracy improved by up to 13% for daily prices and 5.5% for hourly prices.
It didn't matter if the detective was a simple math model or a super-complex AI; the "teamwork" approach helped them all.
The cost to do this "teamwork" was tiny—like adding a few seconds to the computer's thinking time.

Why Should You Care?

Electricity isn't just a number on a screen; it powers our homes, factories, and electric cars.

For the Market: As electricity trading becomes more complex (moving from 24 hourly slots to 96 fifteen-minute slots), getting the price right becomes harder. THieF is a tool that helps traders and grid operators avoid costly mistakes.
For the Future: The paper shows that even the most advanced AI (the "Super-Brain") can be improved by looking at the problem from different angles and forcing those angles to agree.

The Bottom Line

Think of THieF as forcing your forecasters to have a group discussion before making a decision. Instead of letting one person shout the loudest, you listen to everyone, find the common ground, and create a single, super-accurate plan. The paper proves that in the chaotic world of electricity prices, this simple idea of "getting everyone on the same page" saves a lot of money and reduces errors.

Here is a detailed technical summary of the paper "Stealing accuracy: Predicting day-ahead electricity prices with temporal hierarchy forecasting (THieF)".

1. Problem Statement

Operational decisions in wholesale electricity markets require forecasts at varying levels of granularity (e.g., hourly prices vs. daily average or "baseload" prices).

The Conflict: Models trained specifically for hourly products often utilize different information sets than those for daily averages. Consequently, these forecasts frequently lack coherence (i.e., the average of the hourly forecasts does not equal the daily forecast), leading to suboptimal trading and operational decisions.
The Gap: While Temporal Hierarchy Forecasting (THieF) has been applied to load and renewable generation forecasting, it has not yet been rigorously tested for Electricity Price Forecasting (EPF). Furthermore, existing literature focuses on traditional, smaller models, leaving a gap in understanding how THieF performs with modern, large-scale foundation models.
Market Context: The European day-ahead market is shifting toward 15-minute trading intervals, increasing the complexity of trading and the importance of block products (aggregated time periods), making coherent forecasting across time scales critical.

2. Methodology

The study employs a Temporal Hierarchy Forecasting (THieF) framework to reconcile forecasts across different time aggregation levels.

A. Temporal Hierarchy Construction

The authors define a hierarchy based on non-overlapping temporal aggregation of hourly data ( $m=24$ ).

Levels: The hierarchy includes divisors of 24: 1H (hourly), 2H, 3H, 4H, 6H, 8H, 12H, and 24H (baseload).
Summing Matrix ( $S$ ): A matrix is constructed to map hourly observations to aggregated block averages. Unlike standard summing matrices, this study uses mean aggregation (normalizing weights) to ensure the reconciliation process aligns with the statistical properties of the data.
Reconciliation Formula: The base forecasts ( $\hat{P}_d$ ) are reconciled using the optimal combination approach:
$\tilde{P}_d = S(S^T W^{-1} S)^{-1} S^T W^{-1} \hat{P}_d$
Where $W$ is the covariance matrix of the base forecast errors. To handle the high dimensionality ($60 \times 60$ matrix) relative to the sample size, the authors use regularized covariance estimators, specifically the Ledoit and Wolf (2004) constant correlation shrinkage.

B. Base Forecasting Models

To test the versatility of THieF, four distinct model architectures were trained independently for each of the 60 blocks (24 hours $\times$ various block sizes):

Linear Regression (ARX): A parsimonious expert model using lagged prices, load, wind, and commodity futures (coal/gas).
Shallow Neural Networks (NARX): A committee machine of 10 feedforward networks with hyperbolic tangent activation.
Gradient-Boosted Decision Trees (XGB): An ensemble of XGBoost models with Bayesian hyperparameter tuning.
Pretrained Transformer (Mitra): Amazon's AutoGluon "Mitra," a 12-layer, 72-million-parameter tabular foundation model. This is the first application of a pretrained foundation model to EPF and temporal hierarchies, used in a zero-shot mode (no fine-tuning).

C. Data and Experimental Setup

Markets: German (EPEX-DE) and Spanish (OMIE) power markets.
Period: A challenging 4-year out-of-sample test period (2021–2024), covering the post-pandemic era, the Russian invasion of Ukraine, and extreme natural gas price volatility.
Features: 20 input features including lagged prices, load/wind forecasts, and commodity futures.
Preprocessing: Inputs were transformed using the area hyperbolic sine (asinh) function to handle skewness and outliers.

3. Key Contributions

First Application to EPF: This is the first study to apply temporal hierarchy forecasting to electricity price prediction, demonstrating its efficacy beyond load and generation forecasting.
Foundation Model Integration: It introduces the use of a state-of-the-art pretrained transformer (Mitra) for EPF and shows that THieF improves even these sophisticated, zero-shot models.
Robustness Across Regimes: The study validates the method across two distinct markets and a highly volatile 4-year period characterized by structural breaks and extreme price spikes (including negative prices in Germany).
Block Trading Relevance: The results directly address the growing importance of block trading in European markets, showing that reconciliation significantly improves accuracy for larger time blocks.

4. Key Results

The study evaluated performance using Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) over the 4-year test period.

Universal Improvement: THieF improved forecast accuracy for all model architectures and all aggregation levels in both markets.
Magnitude of Gains:
- Hourly (1H): Improvements up to 5.5% (RMSE) for the XGB model in Germany.
- Baseload (24H): Improvements up to 13.4% (RMSE) for the XGB model in Germany.
- General Trend: Accuracy gains generally increase with the size of the time block (e.g., 24H blocks benefit more than 1H blocks).
Model Performance:
- XGB consistently achieved the highest relative gains from reconciliation.
- Mitra (Transformer) also saw significant improvements (e.g., 8.1% MAE gain for 24H in Germany), proving that even foundation models benefit from the implicit combination of information provided by THieF.
Statistical Significance: The Diebold-Mariano test confirmed that reconciled forecasts are statistically significantly more accurate than unreconciled ones at the 1% level for most cases.
Market Differences: Gains were generally higher in the German market (higher volatility) compared to the Spanish market, which entered a calmer regime after 2022.

5. Significance and Implications

Practical Utility: The computational cost of reconciliation is negligible compared to the cost of training the base models. The total computational overhead is approximately 2.5 times the baseline, which is considered minor given the sub-second runtime of standard models.
Decision Alignment: By ensuring that hourly, block, and daily forecasts are coherent, market participants can make better-aligned trading and operational decisions without the risk of conflicting signals from different models.
Future-Proofing: As European markets move to 15-minute trading intervals (96 products/day), the complexity of managing these products will increase. THieF provides a scalable framework to maintain accuracy and coherence across this expanded hierarchy.
Theoretical Insight: The results suggest that even highly complex models (like Transformers) suffer from some degree of misspecification regarding the underlying data generating process. THieF acts as a robust correction mechanism that leverages the strengths of different temporal views to mitigate these errors.

In conclusion, the paper argues that Temporal Hierarchy Forecasting should become a standard component of daily electricity price forecasting pipelines, offering a "free lunch" in terms of accuracy improvement with minimal computational penalty.