🤖 machine learning

Adaptive Online Learning with LSTM Networks for Energy Price Prediction

This paper presents an adaptive online learning framework using LSTM networks with a novel custom loss function and comprehensive feature integration to significantly improve the accuracy of day-ahead electricity price forecasting in the California energy market.

Original authors: Salih Salihoglu, Ibrahim Ahmed, Afshin Asadi

Published 2026-08-13

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Salih Salihoglu, Ibrahim Ahmed, Afshin Asadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess the price of a ticket to a concert that happens every single day, but the price changes every hour based on how many people want to go, how hot the weather is, and how much power the band's sound system needs. This is the chaotic world of electricity markets. Unlike buying a loaf of bread where the price is usually the same, electricity prices are wild, jumping up and down like a rollercoaster because the sun might stop shining (killing solar power) or a heatwave might make everyone turn on their AC at once. To keep the lights on and the bills fair, grid operators and power companies need to predict these prices for the next day, hour by hour. They use "machine learning," which is basically teaching a computer to find patterns in a mountain of past data, kind of like how a detective studies old crime scenes to solve a new case. The goal is to build a crystal ball that doesn't just guess the average price, but understands the crazy spikes and dips before they happen.

This paper is about building a super-smart crystal ball for electricity prices in California using a type of artificial intelligence called an LSTM (Long Short-Term Memory) network. Think of an LSTM as a student with an incredible memory who can remember what happened last week, last month, and even last year, and use that to understand what's happening right now. The researchers, Salih Salihoglu and his team, didn't just let this student study the past; they gave it a special set of rules to learn from. First, they taught it to look at not just past prices, but also the weather and exactly which power plants (solar, wind, gas, nuclear) were running that day. Second, they invented a new "homework grading system" (a custom loss function) that didn't just check if the numbers were close, but also checked if the shape of the price curve looked realistic and smooth, rather than jagged and weird. Finally, instead of letting the student take a test once and then never studying again, they set up a system where the student gets a daily quiz. If the student does better on the quiz after studying a new day's data, they get to update their notes; if they do worse, they stick to their old notes.

The team found that this "adaptive online learning" approach was the winner. Their model, which updates itself carefully every day, was significantly better than models that just sit still and never change (static models) or models that get completely retrained from scratch every day (dynamic models). Specifically, their smart, updating model reduced the average error in its predictions by about 23% compared to the static model. They also discovered that giving the model a list of which energy sources were active (the "generation mix") was a game-changer; without this info, the model was much more confused. Furthermore, their special grading system, which included a penalty for jagged, unrealistic price jumps, helped the model predict the tricky peak hours much more accurately. In short, the paper suggests that the best way to predict these wild electricity prices isn't to build a perfect model once and forget it, but to build a flexible model that learns a little bit every day, checks its own work, and only changes its mind when it's sure it's getting smarter.

Technical Summary: Adaptive Online Learning with LSTM Networks for Energy Price Prediction

Problem Statement

Accurate prediction of day-ahead electricity prices is critical for stakeholders in deregulated energy markets, including grid operators, producers, and consumers. However, forecasting remains challenging due to the high-frequency fluctuations, non-linearity, and volatility inherent in electricity markets. These characteristics are driven by dynamic factors such as weather conditions, demand patterns, and the energy generation mix. Traditional statistical models (e.g., ARIMA, GARCH) often struggle to capture complex temporal dependencies and non-stationary market behaviors, leading to sub-optimal decision-making. Furthermore, static machine learning models fail to adapt to evolving market conditions, while existing online learning approaches may lack rigorous validation mechanisms, making them sensitive to noisy or non-representative data.

Methodology

The study proposes a comprehensive framework for day-ahead electricity price forecasting in the California (CAISO) market, specifically for the NP-15 pricing node. The methodology integrates three core components:

1. Model Architecture

The core predictive engine is a Long Short-Term Memory (LSTM) network designed to forecast a 24-hour vector of electricity prices simultaneously, rather than relying on recursive single-step forecasts.

Input: The model utilizes a sliding window of the last $N$ days of features, combined with same-day known features (e.g., predicted weather, generation mix) for the target day.
Structure: The architecture consists of two stacked LSTM layers (256 units each) with dropout rates of 0.3 to prevent overfitting, followed by a dense output layer that generates the 24-hour price vector.
Features: The input includes historical prices, load demands, natural gas prices, detailed weather data (temperature, wind, humidity, etc.), and the energy generation mix (solar, wind, nuclear, gas, etc.).

2. Custom Loss Function

To enhance prediction robustness and interpretability, the authors introduce a novel custom loss function that combines three components:

Mean Absolute Error (MAE): Minimizes pointwise prediction errors.
Jensen-Shannon Divergence (JSD): Measures the similarity between the probability distribution of the predicted 24-hour price profile and the actual profile. This acts as a shape-based regularizer, ensuring the model captures intra-day trends (peaks and lows) rather than just absolute magnitudes.
Smoothness Penalty (SM): Penalizes abrupt changes between consecutive hourly predictions to ensure realistic, smooth price trajectories.
The total loss is a weighted sum: $Loss = MAE + \alpha \cdot JSD + \beta \cdot SM$ .

3. Adaptive Online Learning Framework

To address concept drift and market volatility, the study implements a validation-based selective online learning mechanism.

Process: As new daily data batches become available, the model evaluates whether updating its parameters improves performance on a validation set.
Selective Update: A temporary parameter update ( $\theta'$ ) is calculated via gradient descent. The update is only accepted if the validation loss decreases by a predefined margin ( $\delta$ ) compared to the current model ( $\theta^*$ ).
Partial Freezing: During updates, the earlier LSTM layers are frozen to preserve learned temporal representations, while only the final dense layer is updated. This strategy mitigates overfitting to short-term noise while allowing adaptation to new trends.

Key Contributions

The paper identifies four primary contributions:

24-Dimensional Vector Prediction: Formulating day-ahead forecasting as a 24-vector prediction task allows the model to explicitly learn intra-day temporal dependencies, avoiding the error accumulation common in recursive single-step approaches.
Novel Custom Loss Function: The integration of MAE with JSD and smoothness penalties creates a loss landscape that encourages realistic, stable, and distributionally accurate price trajectories.
Validation-Based Selective Online Learning: Unlike continuous retraining strategies, this framework updates the model only when new data demonstrably improves generalization performance, offering robustness against noisy or anomalous data.
Systematic Feature Evaluation: The study rigorously evaluates the impact of exogenous variables, specifically weather conditions and energy generation mix data, under both static and adaptive learning settings.

Experimental Results

The framework was evaluated using a dataset spanning 2020 to mid-2024 from the IISE ESD/QCRE/PG&E Energy Analytics Challenge.

Loss Function Impact: The full custom loss function (MAE + JSD + SM) achieved the lowest error metrics. Compared to a standard MSE loss baseline (which yielded an MSE of 315.82), the custom loss reduced MSE by approximately 50% (to 158.76), MAE by 38% (from 12.68 to 7.76), and RMSE by 29% (from 17.77 to 12.60).
Adaptive vs. Static vs. Dynamic:
- The Adaptive Online Learning model outperformed both the Static Model (trained once) and the Dynamic Model (retrained daily from scratch).
- Compared to the Static Model, the Adaptive Online Learning model reduced MSE by ~23.0%, MAE by ~3.4%, and RMSE by ~12.2%.
- The Dynamic Model performed worse than the Static Model, suggesting that frequent full retraining without selective validation leads to overfitting and instability.
Feature Importance:
- Generation Mix: Including fuel mix data significantly improved performance. For the Online Learning model, MSE dropped from 230.38 (without mix) to 122.25 (with mix).
- Same-Day Features: Incorporating same-day operational data (e.g., current weather, load forecasts) drastically reduced errors. The Online Learning model's MSE decreased from 286.42 to 122.25 when same-day features were included.
Benchmark Comparison: The proposed framework outperformed traditional benchmarks including SVR, Decision Trees, Gradient Boosting, ARIMA, SARIMA, and MARIMA. Notably, it achieved an MSE of 122.25, significantly lower than MARIMA (179.47) and ARIMA (396.27).
Statistical Significance: Paired Wilcoxon signed-rank tests and block bootstrap confidence intervals confirmed that the improvements of the Online Learning model over the Static baseline and MARIMA were statistically robust, particularly in MAE and RMSE.

Significance and Claims

The authors claim that this research provides a robust framework for electricity price forecasting that bridges the gap between advanced machine learning techniques and practical market needs. The significance lies in:

Decision Support: Offering stakeholders a reliable tool for navigating dynamic markets, aiding in demand response, optimal bidding, and cost savings.
Methodological Advancement: Demonstrating that combining vector-based forecasting, distribution-aware loss functions, and selective online updates yields superior performance compared to static or blindly adaptive models.
Generalizability: While tested on the California market, the authors note the framework is designed to be applicable to other electricity markets, provided sufficient historical data and relevant exogenous variables are available.

The study concludes that the integration of comprehensive feature sets (specifically generation mix), custom regularization, and adaptive learning strategies is essential for achieving high accuracy and robustness in the face of volatile energy markets.