📈 economics

Imbalance Price Forecasting in the Italian Energy Market Using Deep Neural Networks and Temporal Fusion Transformers

This paper proposes a probabilistic forecasting framework using Temporal Fusion Transformers for the volatile Italian electricity imbalance market, demonstrating that while deterministic accuracy is limited, the model's primary value lies in its ability to provide calibrated risk bands and interpretable insights for effective risk management.

Original authors: Davide Milillo, Valentino Caputi, Rafiq Asghar, Husam S. Samkari, Michele Quercio, Francesco Riganti Fulginei

Published 2026-09-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Davide Milillo, Valentino Caputi, Rafiq Asghar, Husam S. Samkari, Michele Quercio, Francesco Riganti Fulginei

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Electricity is a commodity that cannot be easily stored in vast quantities, meaning the amount generated must match the amount consumed at every single second. In Italy, as in many other nations, this balance is managed through a complex system of markets. Traders buy and sell power in advance for the next day, but the real test comes when the lights are switched on and the actual demand arrives. If a power plant produces too much or too little compared to what was promised, the grid operator must step in to fix the difference, often at a high cost. These costs are known as imbalance prices. Because the grid is a physical network of wires and transformers, local bottlenecks can cause these prices to swing wildly, sometimes spiking to extreme levels in a matter of minutes. Predicting these prices is notoriously difficult, as they are driven by a chaotic mix of weather, renewable energy output, and the physical limits of the transmission lines. For anyone buying or selling electricity, knowing whether these prices will be high or low, and whether they will be positive or negative, is the difference between profit and loss.

A team of researchers from Roma Tre University and the University of Tabuk set out to tackle this problem using a sophisticated type of artificial intelligence called a Temporal Fusion Transformer. Their goal was not simply to guess a single number for what the price would be, but to build a system that could understand the uncertainty inherent in the Italian energy market. They fed the computer model a vast array of information, including historical prices, the amount of electricity being used, how much solar and wind power was available, and the prices set in the day-ahead market. The model was designed to look at this data and forecast what would happen over the next twelve hours, broken down into fifteen-minute intervals. The researchers were particularly interested in whether this advanced technology could finally crack the code of the Italian market, which is known for its extreme volatility and frequent, unpredictable price spikes.

The results of their work revealed a hard truth about the nature of the Italian electricity grid. While the artificial intelligence model performed exceptionally well at predicting the standard day-ahead prices, with errors often less than one euro per unit of energy, it hit a wall when trying to predict the final imbalance prices. In these volatile moments, the model's error rate settled around sixty-two euros per unit, a figure that was only marginally better than much simpler methods. The researchers found that the model could not reliably predict the exact height of price spikes or the precise moment they would occur. In fact, when the model tried too hard to chase these extreme peaks during its training, it actually became less accurate when tested on new data, a phenomenon known as overfitting. The system essentially learned that the market is too chaotic to be predicted with a single, precise number.

However, the study did not end in failure; rather, it led to a shift in how the researchers viewed the value of such a tool. Instead of acting as a crystal ball that gives a single, definitive price, the model proved its worth by acting as a risk manager. It successfully predicted the general direction of the market—whether prices would be high or low, or whether the imbalance would be positive or negative—with an accuracy of about sixty-two to sixty-five percent. More importantly, the model provided a range of possible outcomes, offering a confidence band that showed traders how much risk they were taking. It could identify hours where the market was stable and safe to trade, versus hours where the uncertainty was so high that trading was dangerous.

The researchers also used the model's internal mechanics to understand what was driving the market. By looking at which pieces of data the computer paid attention to, they discovered that the model learned to ignore many of the complex variables it was fed, focusing instead on a few key drivers like the price in the northern zone and the time of year. This transparency turned the "black box" of artificial intelligence into a clear decision-support tool. The system showed that while no one can perfectly predict the chaotic spikes of the Italian grid, traders can still make better decisions by understanding the probability of different outcomes. The study concludes that the true power of this technology lies not in replacing human judgment with a perfect forecast, but in providing a robust, interpretable framework that helps market participants navigate the inherent uncertainty of the energy system with greater confidence.

Technical Summary: Imbalance Price Forecasting in the Italian Energy Market Using Deep Neural Networks and Temporal Fusion Transformers

Problem Statement
The Italian electricity imbalance market is characterized by extreme volatility, non-linear dynamics, and high stochasticity, driven by structural factors such as transmission congestion, the geographical isolation of Sardinia and Sicily, and a unique regulatory framework. These factors render deterministic point forecasting inadequate for operational decision-making. While load forecasting is mature, predicting settlement prices (imbalance prices) remains a complex challenge where traditional statistical and deep learning models often hit a performance ceiling. Existing literature suggests that in the Italian context, even advanced models struggle to achieve Mean Absolute Error (MAE) values below 50–60 €/MWh, with sign prediction accuracy (Long/Short) rarely exceeding 65–70%. Furthermore, standard models (e.g., LSTM, GRU) typically function as "black boxes," offering point estimates without probabilistic confidence intervals or interpretability regarding the market drivers behind the forecast.

Methodology
The authors propose a probabilistic forecasting framework based on the Temporal Fusion Transformer (TFT) tailored to the Italian market's 15-minute resolution and 12-hour multi-horizon prediction requirements.

Architecture: The study utilizes the TFT, which is uniquely suited to handle heterogeneous data streams. The model partitions inputs into:
- Static covariates: Time-invariant features (e.g., day of week, month).
- Past observed inputs: Historical imbalance prices, aggregate grid load, and generation metrics from 20 distinct energy sources.
- Future known inputs: Deterministically available signals prior to market gate closures, including projected load, generation forecasts, and Day-Ahead (MGP) zonal prices.
Training Objective: Unlike standard models optimizing for point accuracy, the TFT is trained via Quantile Loss (Pinball Loss) to output uncertainty intervals at quantiles $q \in \{0.1, 0.5, 0.9\}$ . This provides an 80% confidence interval for risk management rather than a single scalar prediction.
Interpretability Mechanisms: The model employs Variable Selection Networks (VSN) using Gated Residual Networks (GRN) to dynamically suppress non-informative noise and Temporal Multi-Head Attention to identify dominant market drivers across different time scales.
Operational Pipeline: A robust MLOps pipeline was engineered using Google Cloud, ingesting real-time data via Terna and ENTSO-E APIs. The system includes automated validation, handling of asynchronous data latencies, and continuous online training with a 12-hour (48-step) horizon.

Key Results
Experimental results across the Italian market segments reveal a distinct dichotomy between auction markets and the imbalance market:

Auction Markets (PUN, Zonal, XBID): The model performed well, achieving low error margins (e.g., MAE of 4.52 €/MWh for PUN, < 10 €/MWh for Zonal/XBID), demonstrating the architecture's feasibility for standard price forecasting.
Imbalance Price Forecasting:
- Deterministic Ceiling: The TFT achieved an MAE of approximately 62 €/MWh and a sign prediction accuracy of 62–65%. These results are comparable to baselines like LSTM, XGBoost, and even a Naïve model (predicting yesterday's price), confirming a structural accuracy ceiling driven by intrinsic market aleatoric uncertainty rather than model limitations.
- Overfitting vs. Generalization: Initial training without regularization led to high training-set fidelity but catastrophic out-of-sample decay (sign accuracy dropping to ~29%). Through early stopping and dropout, the model converged to a stable "mean-value predictor," sacrificing peak tracking for robust directional accuracy.
- Probabilistic Value: The model's primary utility was found in its probabilistic outputs. The quantile forecasts provided calibrated risk bands, allowing traders to identify low-risk "biddable hours" rather than relying on brittle point estimates.
- Interpretability: Attention maps revealed that the model eventually focused on fundamental drivers like the North zone imbalance price and structural "Year" variables, effectively ignoring spurious correlations (e.g., specific renewable sources) that caused overfitting in earlier iterations.

Significance and Claims
The paper positions its contribution not as a breakthrough in reducing deterministic error, but as a paradigm shift in how forecasting is applied to the Italian imbalance market.

Decision Support over Prediction: The authors argue that the TFT serves as an explainable decision-support system rather than a "price oracle." Its value lies in quantifying risk through uncertainty intervals and providing transparency into market drivers via attention diagnostics.
Probabilistic Necessity: The study empirically demonstrates that in a congestion-driven, zonal market like Italy's, deterministic point predictions face a hard ceiling. The operational objective must be reframed around quantile-based uncertainty modeling to support risk-aware trading strategies.
Regulatory and Structural Context: The work highlights that the performance gap between Italy and other European hubs (e.g., Germany) is due to structural topological and regulatory characteristics (e.g., post-hoc coefficients in settlement equations) rather than algorithmic deficiencies.
Future Outlook: The authors modestly conclude that while the current framework is production-ready, future improvements require richer external datasets and multi-seasonal validation to account for long-term data drifts, rather than simply scaling model capacity.

Technical Summary: Imbalance Price Forecasting in the Italian Energy Market Using Deep Neural Networks and Temporal Fusion Transformers

More like this