⚡ electrical engineering

Efficient reformulations of ReLU deep neural networks for surrogate modelling in power system optimisation

This paper proposes a computationally efficient linear programming reformulation for convexified ReLU deep neural networks that enables their tractable and high-fidelity integration into power system optimization problems, as demonstrated by superior performance in a Danish tertiary capacity market bidding case study compared to existing methods.

Original authors: Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A. R. Liisberg, Julian Lesmos-Vinasco

Published 2026-01-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yogesh Pipada Sunil Kumar, S. Ali Pourmousavi, Jon A. R. Liisberg, Julian Lesmos-Vinasco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-stakes auction for electricity. You are an "aggregator," a middleman who buys flexibility from thousands of homeowners (prosumers) to help balance the power grid. To win, you need to predict exactly how much energy people will shift their usage to save money, based on the price you offer them.

The problem? Human behavior is messy, non-linear, and hard to write down with simple math. To solve this, researchers use Artificial Intelligence (AI), specifically a type called a Deep Neural Network (DNN), to learn these patterns. Think of the AI as a super-smart apprentice who has studied millions of past transactions and can now guess the outcome perfectly.

However, there's a catch: You can't just ask this AI apprentice to sit in the middle of your auction math. The math required to run the auction (optimization) gets stuck in a traffic jam when it tries to process the AI's complex, "black box" logic. It's like trying to drive a Formula 1 car through a narrow, winding village street; the car is too complex for the road.

The Problem: The "Traffic Jam"

Traditionally, to make the AI work with the auction math, researchers had to force the AI into a rigid format using "binary switches" (like turning lights on and off). This works, but it turns the problem into a Mixed-Integer Program (MIP).

The Analogy: Imagine trying to solve a puzzle where every piece can be either "on" or "off." As the puzzle gets bigger (more neurons in the AI), the number of possible combinations explodes. The computer gets overwhelmed, taking hours or days to find a solution, or it gives up entirely.

The Existing "Fixes" and Why They Fail

Researchers tried two other ways to smooth out the road:

Penalty Methods (PCAR/PCTAR): These try to trick the computer by adding a "fine" (penalty) if the AI guesses wrong.
- The Flaw: It's like telling a driver, "If you go off-road, you pay a fine." But the driver might decide it's cheaper to pay the fine and still crash. These methods are sensitive; if you set the fine too low, the AI ignores it. If too high, the math breaks. The results are often unreliable.
Piecewise Linearization (PWL): This breaks the AI's curve into straight lines.
- The Flaw: It's accurate, but it's like trying to draw a smooth circle using only straight Lego bricks. You need so many bricks that the model becomes huge and slow to compute.

The Paper's Solution: The "Convexified" Shortcut

The authors propose a clever new way to reformat the AI so it fits on the road without getting stuck. They call it a Convexified ReLU DNN.

Here is the magic trick:

The Goal: In their specific auction problem, the goal is to minimize costs (spend as little as possible to buy flexibility).
The Constraint: The researchers force the AI's internal weights (the "knobs" the AI turns) to be positive numbers only for most of its layers.
The Result: By forcing these knobs to be positive, the AI's behavior changes from a jagged, unpredictable mountain range into a smooth, bowl-shaped valley.
- The Analogy: Imagine the AI is a ball rolling down a hill. In the old version, the hill had hidden traps and cliffs (non-convex). In the new version, they paved the hill into a perfect, smooth slide (convex). Because the goal is to find the lowest point (minimize cost), the ball will naturally roll to the bottom without getting stuck.

Because the shape is now a smooth bowl, the computer doesn't need to flip switches or pay fines. It can use Linear Programming (LP), which is like a super-fast, straight-line highway. The computer zooms through the problem in milliseconds instead of hours.

What They Found

The team tested this on a real-world scenario: the Danish electricity market. They compared their new "smooth slide" method against the old "traffic jam" methods and the "fine-based" tricks.

Speed: Their method was hundreds to thousands of times faster than the traditional AI methods. It solved the problem in milliseconds.
Accuracy: It was almost as good as the most accurate (but slow) methods. It didn't make the "crashing" mistakes that the "fine-based" methods did.
Reliability: Unlike the other methods, their solution didn't break when they made the AI bigger or more complex. It stayed fast and reliable.

The Catch (The Fine Print)

This shortcut only works if two conditions are met:

You are trying to minimize the AI's output (like minimizing cost).
You are willing to train the AI with the rule that its internal knobs must be positive (which limits the AI slightly, but in a way that still captures the essential behavior needed for this specific problem).

The Bottom Line

This paper doesn't invent a new AI; it invents a new translation tool. It takes a complex, slow-to-solve AI model and translates it into a simple, fast-to-solve math problem, provided you are looking for the lowest cost. This allows power grid operators to use smart AI predictions in real-time auctions without waiting days for the computer to finish the math.

Technical Summary: Efficient Reformulations of ReLU Deep Neural Networks for Surrogate Modelling in Power System Optimisation

Problem Statement
The decarbonisation of power systems has introduced complex, nonlinear interactions driven by distributed energy resources, which are difficult to capture using conventional optimisation models. While Machine Learning (ML) based surrogate modelling, particularly using Rectified Linear Unit (ReLU) Deep Neural Networks (DNNs), offers a promising solution for representing these behaviors, integrating ReLU DNNs directly into optimisation frameworks presents significant challenges. Standard approaches using Mixed-Integer Programming (MIP) reformulations (e.g., Big-M constraints) introduce binary variables for every neuron, leading to computational intractability as network depth and width increase. Conversely, existing Linear Programming (LP) relaxations, such as Penalised Convex Area Relaxation (PCAR) and Penalised Convex Triangle Area Relaxation (PCTAR), improve tractability but rely heavily on heuristic penalty terms. These penalty-based methods often lack guaranteed tightness and robustness, resulting in sensitivity to hyperparameter tuning and potential solution quality degradation.

Methodology
The paper proposes a novel LP reformulation for a specific class of ReLU DNNs, termed convexified (cvxd) ReLU DNNs. The core mathematical insight relies on Theorem 2.1, which establishes that if a ReLU DNN's output is minimised within an optimisation problem, and the weight matrices of all layers from the second layer onwards are constrained to be element-wise non-negative ( $W_l \geq 0$ for $l \geq 2$ ), the non-convex ReLU activation functions in those layers can be replaced by linear inequalities without losing tightness.

Specifically:

Convexification: The method constrains the weight matrices of hidden layers (from layer $k$ to the output) to be non-negative. This structural constraint ensures that the minimisation objective propagates through the network, allowing the replacement of $\max(\cdot, 0)$ constraints with linear inequalities ( $h_l \geq W_l h_{l-1} + b_l$ and $h_l \geq 0$ ).
Application Context: The method is applied within an Optimisation with Constraint Learning (OCL) framework. The authors learn a prosumer responsiveness function (a nonlinear, nonconvex saturation curve) using a ReLU DNN and embed it into an energy aggregator's profit maximisation problem for the Danish tertiary capacity market (mFRR).
Benchmarking: The proposed reformulation is evaluated against State-of-the-Art (SOTA) alternatives:
- MIP-based: Standard Big-M reformulation using the Gurobi ML package.
- LP-based: PCAR and PCTAR relaxations.
- Conventional: Piecewise Linearisation (PWL).

Key Contributions

Tractable LP Reformulation: The paper introduces a method to convert ReLU DNNs into tight LP formulations by enforcing non-negative weights on layers subsequent to the first. This eliminates the need for binary variables, significantly reducing computational complexity compared to MIP approaches.
Guaranteed Tightness: Unlike penalty-based relaxations (PCAR/PCTAR), the proposed method guarantees tightness and accuracy across various network architectures and input conditions, provided the minimisation and non-negative weight conditions are met.
Systematic Evaluation: The study provides a comprehensive benchmark across multiple neural network architectures (varying width and depth) and market price scenarios (low, medium, high), demonstrating the scalability and reliability of the approach.

Results
The evaluation was conducted using a case study of an aggregator bidding in the Danish mFRR market, utilizing 30,000 training data points and testing across 30 market scenarios.

Computational Performance: The proposed LP reformulation significantly outperformed MIP-based (Gurobi ML) and PWL approaches in terms of speed. While Gurobi ML and PWL often required seconds to hours (or failed to converge within time limits for wider networks), the proposed method solved problems in milliseconds (e.g., 34 ms for high-price scenarios vs. 1–3 seconds for others).
Solution Quality: The proposed method achieved realised profits comparable to the best-performing methods (PWL and MIP) across all scenarios, typically within 1% of the top performer. In contrast, PCAR and PCTAR frequently resulted in economic losses, particularly in low-price scenarios, due to the mis-estimation of flexibility costs caused by their penalty-driven structure.
Scalability: The proposed method maintained consistent performance and tightness as network width and depth increased. In contrast, Gurobi ML exhibited severe scalability issues, with runtime spiking and MIP gaps widening significantly for wider networks (e.g., 400 neurons). PCAR and PCTAR showed inconsistent accuracy and high sensitivity to penalty parameter tuning.
Accuracy: The proposed model consistently achieved Root Mean Square Error (RMSE) values close to the training RMSE of the underlying cvxd DNN, confirming the tightness of the reformulation.

Significance and Claims
The paper claims that convexified ReLU DNNs, when paired with the proposed LP reformulation, offer a scalable and reliable methodology for integrating learned surrogate models into power system optimisation. The significance lies in bridging the gap between the expressive power of deep learning and the computational requirements of large-scale optimisation.

The authors emphasise that while the method requires the network output to be minimised and imposes non-negativity constraints on weights (limiting the class of learnable functions to convex and monotonically increasing relationships), it successfully captures complex flexibility cost functions in the tested case study. The work positions itself as a general methodological advancement that avoids the heuristic tuning and lack of tightness guarantees associated with existing LP relaxations, making it suitable for applications such as generator cost modelling, CO2 emissions estimation, and chance-constrained formulations where the objective involves minimisation. The authors note that the primary focus is the demonstration of the reformulation's properties rather than the specific aggregator application, which serves as a vehicle for validation.