Regime-Aware Conditional Neural Processes with Multi-Criteria Decision Support for Operational Electricity Price Forecasting

This paper proposes a Regime-Aware Conditional Neural Process (R-NP) model that combines Bayesian regime detection with localized neural processes to forecast German electricity prices, demonstrating through multi-criteria TOPSIS analysis that it offers the most balanced and preferred solution for operational battery storage optimization across 2021–2023, outperforming both deep neural networks and Lasso-based models despite raw accuracy trade-offs.

The Gist

Imagine you are the captain of a ship (a battery storage system) trying to navigate a very stormy ocean (the electricity market). Your goal is to make the most money possible by buying fuel (electricity) when it's cheap and selling it when it's expensive.

The problem? The ocean doesn't behave the same way every day. Sometimes it's calm and predictable; other times, it's chaotic with sudden, massive waves (price spikes) caused by things like wind, sun, or new government rules.

This paper introduces a new, super-smart navigation system called Regime-Aware Conditional Neural Processes (R-NP) to help captains like you. Here is how it works, broken down into simple concepts:

1. The Problem: The Ocean Changes Its "Mood"

In the past, electricity prices were like a steady river. You could predict the flow easily. But now, with so much wind and solar power, the river turns into a chameleon. It changes its "mood" or Regime constantly:

• Regime A: Calm, sunny days with low prices.
• Regime B: Stormy, windy days with wild price swings.
• Regime C: A sudden crisis where prices shoot up to the moon.

Old navigation systems (like standard computer models) try to learn one rule for the whole ocean. They get confused when the mood shifts, leading to bad decisions. It's like trying to drive a car using a map for a desert while you are suddenly in a jungle.

2. The Solution: A "Mood Detective" and Specialized Guides

The authors built a two-part system to solve this:

Part A: The Mood Detective (DS-HDP-HMM)
First, the system acts like a detective. It looks at the history of electricity prices and asks: "What is the ocean's mood right now?"

• It doesn't guess how many moods there are; it figures it out automatically.
• It notices patterns: "Ah, for the last three days, we've been in a 'High Volatility' mood. Now, we've shifted to a 'Stable' mood."
• Crucially, it knows that some moods last a long time (sticky), while others are just fleeting flashes.

Part B: The Specialized Guides (Conditional Neural Processes)
Once the detective identifies the current mood, the system doesn't use a generic guide. Instead, it calls upon a specialist guide who only knows how to navigate that specific mood.

• If it's a "Stormy" mood, it asks the Storm Guide.
• If it's a "Calm" mood, it asks the Calm Guide.
• These guides are trained specifically on data from those specific times, so they are experts at predicting what happens next in that specific scenario.

Finally, the system combines the advice from all the guides, weighted by how likely each mood is to happen, to give you the best possible forecast.

3. The Twist: Being "Right" Isn't Always "Profitable"

Here is the most interesting part of the paper. The authors tested their system against two other popular navigation tools:

1. The Math Whiz (LEAR): Very good at simple, straight-line predictions.
2. The Deep Learner (DNN): A complex AI that tries to learn everything at once.

They found something surprising: The Math Whiz often made more money than the fancy AI, even if the AI's raw price predictions were slightly more accurate.

Why?
Imagine you are betting on a horse race.

• The Math Whiz might be slightly wrong about the exact time the horse finishes, but it correctly identifies which horse will win. You make money.
• The Fancy AI might predict the exact finish time perfectly, but it gets confused about which horse is the favorite, so you bet on the wrong one. You lose money.

In the electricity market, being "perfectly accurate" on the price doesn't matter as much as knowing the key turning points (when to buy and sell). A model can be slightly "wrong" on the numbers but "right" on the strategy.

4. The Final Verdict: The "All-Rounder" Score

Because there is no single "best" model (one is better at profit, another at safety, another at accuracy), the authors used a scoring system called TOPSIS.

Think of TOPSIS as a Talent Show Judge.

• It doesn't just look at who sings the highest note (accuracy).
• It looks at who sings the best song for the audience (profit), who stays on pitch during a storm (risk management), and who is the most consistent (reliability).

The Results:

• In 2021, the Math Whiz (LEAR) won the talent show.
• But in 2022 and 2023 (when the market got more chaotic), the authors' new system (R-NP) won the top spot.

The Big Takeaway

The paper teaches us that in a complex, changing world, you don't just need a model that is "smart." You need a model that is adaptable.

By first detecting the "mood" of the market and then using a specialist guide for that specific mood, the new system makes better real-world decisions. It proves that for battery owners and energy companies, the best forecast isn't always the one with the lowest error rate on paper; it's the one that helps you make the most money in the real world.

Technical Summary

1. Problem Statement

The paper addresses the challenge of forecasting electricity prices in markets with high renewable energy penetration (specifically the German market). Key difficulties include:

• Non-stationarity and Regime Changes: Electricity prices exhibit structural shifts due to weather, regulatory changes, and renewable intermittency, causing traditional models to fail.
• The Accuracy-Utility Gap: High raw prediction accuracy (low RMSE/MAE) does not necessarily translate to optimal operational outcomes (e.g., battery storage profit maximization or cost minimization). A model might have slightly higher errors but correctly identify critical price turning points, leading to better economic decisions.
• Uncertainty Quantification: Many deep learning models provide deterministic point forecasts, lacking the probabilistic uncertainty estimates required for risk-sensitive financial decision-making.
• Evaluation Complexity: Selecting the "best" model requires balancing conflicting criteria: prediction accuracy, risk management, and operational profitability across different strategies.

2. Methodology

The authors propose a hybrid framework called Regime-Aware Neural Processes (R-NP), which integrates unsupervised regime detection with probabilistic forecasting and multi-criteria decision making.

A. Regime Detection (DS-HDP-HMM)

Instead of assuming a fixed number of market states, the authors use a Disentangled Sticky Hierarchical Dirichlet Process Hidden Markov Model (DS-HDP-HMM).

• Non-parametric: It automatically infers the number of regimes from data without pre-specification.
• Sticky: It encourages state persistence (regimes last for a duration) rather than rapid switching.
• Disentangled: It separates the control of self-transition probabilities (persistence) from the transition diversity between different regimes, allowing for more flexible modeling of market dynamics.
• Process: The model segments historical price data into latent regimes. A separate MLP classifier is then trained to map input features (prices, load, renewable generation) to these regime labels for future prediction.

B. Regime-Specific Forecasting (Conditional Neural Processes - CNP)

Once regimes are identified, the framework trains an independent Conditional Neural Process (CNP) for each regime.

• Mechanism: CNPs are meta-learning models that learn a distribution over functions. They take a "context set" (historical data points) and a "target input" to predict a probability distribution (mean and variance) for the future price.
• Advantage: By training separate CNPs for each regime, the model avoids "mode averaging" (where a global model smooths out distinct behaviors). It captures regime-specific dynamics and provides inherent uncertainty quantification (predictive variance).

C. Probabilistic Aggregation

For a new prediction, the system:

1. Predicts the probability of being in each regime using the MLP classifier.
2. Queries the corresponding regime-specific CNP.
3. Aggregates the outputs into a mixture of Gaussians, weighted by the regime probabilities. This yields a final forecast with a combined mean and total variance (accounting for both within-regime and between-regime uncertainty).

D. Operational Optimization & MCDM

The forecasts are fed into four distinct battery storage optimization strategies:

1. Case I: Pure price arbitrage (maximize profit).
2. Case II: Arbitrage with risk aversion (penalizing trades during high forecast uncertainty).
3. Case III: Grid support (integrating renewable generation and residual load objectives).
4. Case IV: Cost minimization for a fixed load profile.

To evaluate the models, the authors employ TOPSIS (Technique for Order Preference by Similarity to Ideal Solution). This Multi-Criteria Decision Making (MCDM) method ranks models based on a composite score derived from:

• Prediction accuracy metrics (RMSE, MAE, MAPE, SMAPE).
• Operational performance (Realized profit/cost vs. Perfect Foresight benchmark).
• Uncertainty calibration (Prediction Interval Coverage Probability).

3. Key Contributions

1. Novel Architecture: The integration of DS-HDP-HMM for unsupervised regime discovery with CNPs for regime-specific probabilistic forecasting. This addresses the non-stationarity of electricity prices better than global deep learning models.
2. Operational Validation: Moving beyond standard statistical metrics to evaluate models based on their ability to drive optimal battery storage strategies (arbitrage, risk management, grid services).
3. MCDM Framework: Demonstrating that raw accuracy is insufficient for model selection. The use of TOPSIS reveals that a model with slightly higher raw errors can be operationally superior if it better captures regime dynamics or uncertainty.
4. Uncertainty Quantification: Providing a rigorous probabilistic framework that allows for risk-averse decision-making, a feature often missing in standard DNN or Lasso-based models.

4. Results

The study compares R-NP against a Deep Neural Network (DNN) and a Lasso Estimated Autoregressive (LEAR) model using German electricity market data (2021–2023).

• Prediction Accuracy:
  • 2021 (Low Variance): DNN slightly outperformed R-NP in some metrics (MAPE), likely due to over-regularization in R-NP for stable regimes.
  • 2023 (High Variance): R-NP achieved the lowest MAE and RMSE, demonstrating superior adaptation to volatile market conditions.
  • Uncertainty: R-NP provided well-calibrated prediction intervals (high PICP), whereas DNN and LEAR lacked native uncertainty quantification.
• Operational Performance:
  • LEAR often yielded the highest absolute profits or lowest costs in specific years (e.g., 2021), suggesting its linear structure was robust for that specific market state.
  • DNN showed exceptional performance in specific cost-minimization contexts (Case IV, 2022) but struggled in others.
  • R-NP showed the most consistent performance across different years and strategies.
• TOPSIS Ranking:
  • 2021: LEAR was ranked highest.
  • 2022 & 2023: R-NP emerged as the top-ranked model.
  • Conclusion: R-NP provided the most balanced solution, effectively navigating the trade-off between prediction accuracy and operational robustness across changing market conditions.

5. Significance

This paper makes a significant contribution to energy economics and machine learning by:

• Bridging the Gap: It explicitly demonstrates that the "best" forecasting model depends on the downstream application. A model optimized for raw error minimization may not be the best for profit maximization.
• Handling Complexity: It offers a scalable solution for the non-stationary, regime-switching nature of modern renewable-heavy electricity markets.
• Decision Support: By integrating MCDM (TOPSIS), it provides a transparent, quantitative framework for stakeholders (utilities, grid operators, investors) to select forecasting models that align with their specific risk appetites and operational goals, rather than relying solely on statistical error metrics.

The proposed R-NP framework is presented as a robust, uncertainty-aware tool essential for the efficient operation of Battery Energy Storage Systems (BESS) in the transition to decarbonized energy grids.