💰 quantitative finance

A Market-Rule-Informed Neural Network for Efficient Imbalance Electricity Price Forecasting

This paper proposes a market-rule-informed neural network framework that embeds imbalance price formation rules into a neural latent space to achieve accurate, efficient, and robust electricity price forecasting with fewer parameters and faster training than generic deep learning baselines.

Original authors: Runyao Yu, Julia Lin, Derek W. Bunn, Jochen Stiasny, Wentao Wang, Yujie Chen, Tara Esterl, Peter Palensky, Jochen L. Cremer

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Runyao Yu, Julia Lin, Derek W. Bunn, Jochen Stiasny, Wentao Wang, Yujie Chen, Tara Esterl, Peter Palensky, Jochen L. Cremer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Predicting the "Surprise" Price

Imagine the electricity market is like a massive, high-stakes game of poker played by power plants and battery owners. There are three main rounds:

Day-Ahead: You bet on how much electricity you need tomorrow.
Intraday: You adjust your bet as the day goes on.
Balancing (The "Imbalance" Market): This is the final round where the game master (the grid operator) settles the score. If you bet you'd send 100 units of power but only sent 90, you have an "imbalance." You have to pay a penalty price to fix it.

The problem? This penalty price is wildly unpredictable. It can jump from normal to astronomical in seconds. It's not just random chaos, though; it follows a very specific, written rulebook (the "Market Rules").

The Problem with Current AI

Most AI models trying to predict this price are like students who are told to memorize the answers to a math test without ever being taught the formulas. They look at past prices and try to guess the next one using pure pattern recognition (Deep Learning).

The Flaw: This is inefficient. The AI has to "re-learn" the math rules every time it trains, which takes a lot of computer power and memory.
The Data Trap: Sometimes, the AI is fed a "summary" of the past (e.g., "The price was $100"). But different situations can lead to the same $100 price. If the AI only sees the summary, it loses the context of why the price was $100, making it bad at guessing what happens next.

The Solution: MRINN (The "Rule-Book" AI)

The authors built a new AI called MRINN (Market-Rule-Informed Neural Network). Think of it as a student who is given the actual math textbook (the market rules) and a smart tutor (the neural network).

Instead of letting the AI guess the rules from scratch, the researchers hard-coded the market rules directly into the AI's brain.

Here is how they did it, using a simple analogy:

1. The "Secret Sauce" Ingredients

The final price is calculated using three main ingredients:

The Energy Cost: How much it cost to fix the imbalance using batteries or generators.
The Market Price: What the regular electricity price was at that moment.
The Scarcity Penalty: A "panic tax" that kicks in if the grid is running dangerously low on power.

The rulebook says: "Take the highest or lowest of these three ingredients depending on whether we have too much or too little power."

2. The "Translation" Problem

The AI speaks a different language (mathematical vectors) than the rulebook (if/then logic). You can't just paste a rulebook into a neural network; the network gets confused by "If this, then that" logic.

So, the authors built special translators for the AI:

The "Soft" Max/Min: Instead of a hard "Choose the biggest number," they taught the AI a smooth, gentle version of picking the biggest number so it can learn without breaking.
The "Soft" Sign: Instead of a sharp "Is it positive or negative?", they used a smooth curve that gently leans toward positive or negative.
The "Soft" If/Else: Instead of a hard switch that cuts off information, they used a "dimmer switch" that gradually fades between options.

By translating the rigid rules into "soft" math, the AI can understand the rules while it is learning from the data.

Why This is a Game Changer

The paper tested this new AI against the old "pure pattern" AIs and found some surprising results:

It's Smarter with Less Memory: The new AI is tiny. It has only about 1,800 parameters (think of these as the AI's "brain cells"). The other advanced AIs had 10,000 to 28,000 brain cells.
- Analogy: It's like solving a complex puzzle with a small, perfectly shaped key, whereas the other AIs are trying to force a giant, heavy key into the lock.
It's Faster: Because it's smaller, it trains in seconds and runs instantly. This is crucial for real-time trading where decisions happen in milliseconds.
It Doesn't Forget Context: By feeding the AI the raw data (the specific ingredients) and the rules, it avoids the "Data Trap." It knows that a $100 price caused by a battery failure is different from a $100 price caused by a sunny day.
It Handles Missing Data Well: In the real world, data sometimes arrives late (like a delayed text message). The authors tested what happens when the AI is missing certain ingredients. They found that if you remove the "Scarcity" rule, the AI gets confused about extreme price spikes. If you remove the "Energy Cost" rule, it gets bad at predicting average prices. This proves that all three rules are necessary to get the full picture.

The Bottom Line

The paper argues that in fields like electricity trading, you shouldn't just throw a "black box" AI at the problem. Instead, you should teach the AI the rules of the game and let it use its pattern-matching skills to fill in the gaps.

By combining the structure of the market rules with the flexibility of deep learning, they created a model that is:

More accurate at predicting wild price swings.
Much cheaper to run (less computer power).
Better suited for real-world industrial use where data might be messy or delayed.

In short: They didn't just build a better guesser; they built a student who actually studied the rulebook.

Technical Summary: A Market-Rule-Informed Neural Network for Efficient Imbalance Electricity Price Forecasting

Problem Statement

Accurate and efficient forecasting of imbalance electricity prices is critical for industrial energy trading systems, particularly as battery assets and automated bidding pipelines increasingly participate in balancing markets. However, real-time forecasting faces three primary challenges:

Nonlinear Market Rules: Imbalance prices are determined by transparent but complex, piecewise, and nonlinear settlement rules that map system signals (e.g., imbalance volumes, activated reserves) to prices.
Data Heterogeneity and Latency: Forecasting requires synchronizing heterogeneous signals (system, market, and reserve data) under strict temporal constraints, often complicated by communication delays, publication lags, and measurement outages.
Information Loss in Compression: Existing deep learning approaches often treat imbalance prices as generic time-series targets. A critical issue identified is the non-invertibility of the pricing function: compressing raw features ( $F_t$ ) into a lagged price ( $P_t$ ) discards predictive information because multiple distinct raw feature states can map to the same price, making future predictions indistinguishable if only the compressed price is used as input.

Methodology

The authors propose the Market-Rule-Informed Neural Network (MRINN), a framework that embeds known imbalance price formation rules directly into the latent space of an expressive neural network.

Core Architecture

Input Processing: Raw signals (e.g., aFRR/mFRR volumes, intraday prices, system imbalance) are projected into a latent vector space rather than being compressed into a scalar price.
Differentiable Rule Injection: The deterministic pricing rules (Equations 1–16 in the paper) are implemented as differentiable operators within the latent space. To handle non-differentiable or discontinuous operations inherent in market rules, the authors introduce specific approximations:
- Min/Max: Replaced with softplus functions to ensure smooth gradient flow.
- Absolute Value: Approximated using $\sqrt{a^2 + \epsilon}$ .
- Sign Function: Approximated using the hyperbolic tangent (tanh).
- Division: Stabilized by adding a small constant to the denominator.
- Conditional Logic (If-Else): Replaced with a soft selection mechanism using softmax weights to create a differentiable branching structure.
- Constants: Predefined market constants are transformed via RobustScaler to match the latent space scale.
Forecasting Head: The model utilizes a hierarchical quantile head to produce probabilistic forecasts. This design enforces non-negative increments between quantiles (e.g., median to upper/lower bounds), guaranteeing zero quantile crossing and improving reliability.
Training Objective: The model is optimized using Average Quantile Loss (AQL).

Experimental Setup

Dataset: 15-minute resolution data from the Austrian balancing market (2022–2026).
Baselines: The model is benchmarked against:
- Naïve models: Persistence (latest price) and intraday indices.
- Lagged-Price-Based models: Linear Quantile Regression (LQR), XGBoost, and MLP using only past prices.
- Raw-Feature-Based models: Advanced deep learning architectures (AttnBiLSTM, iTransformer, PatchTST, TimesNet, TimeXer) using raw signals.
Evaluation: A 3-fold temporal cross-validation protocol covering a full year, with metrics including AQL, Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Quantile Crossing Rate (AQCR).

Key Results

Performance vs. Baselines:
- MRINN achieved an AQL of 20.70, outperforming the best naïve baseline (25.44) by 18.63% and the best lagged-price-based baseline (MLP, 22.51) by 8.04%.
- In point forecasting, MRINN reduced MAE to 49.36 and RMSE to 277.33, outperforming all other models.
- Crucially, MRINN achieved 0.00% quantile crossing, whereas purely data-driven raw-feature baselines exhibited crossing rates between 6.58% and 8.53%.
Parameter Efficiency:
- MRINN requires only 1,817 trainable parameters (~400 KB model size).
- This is significantly smaller than generic deep learning baselines (e.g., AttnBiLSTM with 28.3k parameters, TimeXer with 21.7k), which often require hundreds of megabytes when scaled.
- Training time is approximately 40 seconds on both CPU and GPU, demonstrating high computational efficiency.
Ablation and Sensitivity:
- Component Contribution: Removing any of the three price components (Balancing-Energy, Market-Reference, Scarcity-Function) degraded performance. The Scarcity component was most critical for uncertainty modeling (AQL), while the Balancing-Energy component was most critical for point accuracy (MAE).
- Scaling Laws: Short-term forecasting (horizon < 180 min) performed best with short input windows (0–60 min), while longer horizons benefited from longer input windows (1440 min), suggesting a trade-off between recent noise and long-term seasonal context.

Significance and Claims

The paper argues that market-rule priors and expressive neural networks should be jointly used for industrial energy trading applications. The key contributions and claims are:

Addressing Non-Invertibility: The study demonstrates that compressing raw signals into lagged prices leads to information loss. By retaining raw features and embedding rules, MRINN preserves predictive information that purely data-driven models must relearn.
Resource Efficiency: The integration of structural inductive bias (market rules) allows the model to achieve competitive accuracy with substantially fewer parameters and shorter training times than generic deep learning baselines. This makes the approach suitable for resource-constrained deployments (e.g., site-server or battery asset management) where memory and latency are limited.
Operational Robustness: The framework provides actionable guidance for real-world deployment, showing how model performance scales with input length and forecasting horizon, and how it behaves when specific signals are missing.
Probabilistic Reliability: The hierarchical quantile head ensures physically consistent probabilistic forecasts (no quantile crossing), a critical requirement for risk management in energy trading.

The authors conclude that while the specific rules are market-dependent, the methodology of embedding differentiable operators into latent spaces offers a path toward "computationally sustainable" forecasting in operational energy systems. Future work is suggested to investigate the reusability of these operator blocks across different market frameworks.