⚡ electrical engineering

Computing an Optimal Control Policy for an Energy Storage

This paper introduces StoDynProg, a generic Stochastic Dynamic Programming library for solving optimal control problems in renewable energy management, and demonstrates its effectiveness by deriving a stochastic model and computing an optimal policy to smooth the power output of an Ocean Wave Energy Converter.

Original authors: Pierre Haessig, Thibaut Kovaltchouk, Bernard Multon, Hamid Ben Ahmed, Stéphane Lascaud

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 3.0

Original authors: Pierre Haessig, Thibaut Kovaltchouk, Bernard Multon, Hamid Ben Ahmed, Stéphane Lascaud

Original paper licensed under CC BY 3.0 (http://creativecommons.org/licenses/by/3.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Taming the Wild Ocean

Imagine you are trying to pour water from a very wobbly, shaking bucket (the ocean wave energy) into a delicate glass (the electricity grid). If you just pour directly, the water will splash everywhere, overflowing the glass or leaving it half-empty. This is the problem with renewable energy: the wind and waves are unpredictable, but the power grid needs a steady, smooth flow.

To fix this, the authors propose using a "buffer" or a "reservoir" (an energy storage system, like a giant battery or super-capacitor). This reservoir catches the extra water when the waves are huge and releases water when the waves are small, smoothing out the flow into the glass.

The paper asks a specific question: How do we control this reservoir perfectly? Should we just use a simple rule (like "if the bucket is full, pour more")? Or can we use a smart computer brain to figure out the best possible way to manage the water at every single second?

The Problem: The Ocean is Chaotic

The authors looked at a specific machine called SEAREV, which is a giant floating pendulum that swings with the waves to generate electricity.

The Issue: The power it generates is wild. It spikes and dips irregularly because the ocean is random.
The Goal: Smooth out these spikes so the power sent to the grid is steady.
The Constraint: The storage system has a limit. It can only hold a certain amount of energy (like a 10 MJ battery). If you fill it up too fast, it overflows; if you drain it too fast, it runs dry.

The Solution: A "Smart Brain" (Stochastic Dynamic Programming)

The authors created a software library called StoDynProg (think of it as a specialized calculator for decision-making). They used a method called Stochastic Dynamic Programming (SDP).

The Analogy: The Chess Player
Imagine you are playing chess, but the board changes randomly every turn (like the ocean waves). You want to make the best move now, but you also have to think about what might happen 10 turns from now.

Heuristic (Simple Rule): A beginner player might just say, "Always capture the nearest piece." This works okay, but it's not perfect.
Optimal Control (The Smart Brain): The SDP method is like a grandmaster who simulates thousands of possible futures. It looks at the current state (how much energy is in the battery, how fast the pendulum is swinging) and calculates the exact best move to minimize chaos over the long run.

How They Made It Work

To teach the computer how to predict the ocean, they had to do some detective work:

Modeling the Chaos: The raw data of the wave machine was too messy to predict directly. So, they broke it down into simpler parts: the speed of the pendulum and its acceleration. They found that while the power is wild, the speed of the pendulum follows a predictable pattern (like a bell curve).
The "Memory" Trick: To predict the future, the computer needs to remember the past. They gave the computer a "memory" of the last few seconds of speed and acceleration. This allowed the computer to treat the random ocean as a predictable "Markov process" (a system where the future depends only on the present state).
The Algorithm: They used an algorithm called Policy Iteration.
- Step 1: Start with a simple, dumb rule (the "linear policy").
- Step 2: Let the computer simulate the ocean thousands of times.
- Step 3: The computer says, "Hey, if the battery is full and the wave is huge, don't just dump the power. Hold back a little."
- Step 4: Repeat this until the rule becomes perfect.

The Results: Smarter Smoothing

When they tested their "Smart Brain" against the "Simple Rule":

The Simple Rule: It smoothed the power okay, but it didn't use the battery efficiently. It often left the battery half-full when it could have been full, or drained it too quickly.
The Smart Brain: It reduced the "bumps" in the power flow by about 20% compared to the simple rule.
- It knew exactly when to charge and when to discharge based on the speed and acceleration of the machine, not just the current power level.
- It learned to "save" energy for the next big wave spike.

The Catch: The "Curse of Dimensionality"

The paper mentions a major hurdle called the Curse of Dimensionality.

The Metaphor: Imagine trying to map every possible combination of weather, traffic, and driver mood to predict a commute. If you add just one more variable (like "time of day"), the number of combinations explodes.
The Reality: The computer had to calculate the best move for every possible combination of:
1. Battery level.
2. Pendulum speed.
3. Pendulum acceleration.
Because the number of combinations grows so fast, they had to keep the model simple (only 3 variables). If they tried to add more details, the computer would take years to solve the problem. This is why they spent so much time simplifying the wave model in the first place.

Conclusion

The authors built a tool (StoDynProg) that can solve complex "best decision" problems for energy systems. By using this tool on a wave energy machine, they proved that a mathematically optimized control strategy is significantly better than a simple, guesswork-based strategy. It makes the energy flow smoother, which is better for the electrical grid, all while using the same amount of battery storage.

In short: They taught a computer to play the "ocean game" perfectly, resulting in a much steadier stream of electricity than a human could manage with simple rules.

Technical Summary: Computing an Optimal Control Policy for an Energy Storage

Problem Statement
The paper addresses the challenge of managing power variability in renewable energy sources, specifically Ocean Wave Energy Converters (WECs) like the SEAREV device. Unlike conventional generation, wave energy output is highly stochastic and fluctuates rapidly (e.g., with a ~1.5s period), posing stability risks to the electrical grid. The authors frame the problem as an Optimal Control task: determining a control policy for an Energy Storage System (ESS) that smooths the power injected into the grid ( $P_{grid}$ ) while respecting the physical constraints of the storage capacity ( $E_{sto}$ ).

The objective is to minimize a cost function $J$ representing the average variability (penalty) of the injected power over an infinite horizon. The system is modeled as a stochastic dynamic optimization problem where decisions at each time step are coupled by the state of stored energy. A key prerequisite for applying Dynamic Programming (DP) is that the system dynamics must be Markovian; however, raw wave power time series are not inherently Markovian, necessitating a transformation of the input data.

Methodology
The authors propose a three-stage methodology combining time series analysis, stochastic modeling, and numerical optimization via a custom Python library.

Stochastic Modeling of Wave Production:
- Direct modeling of power ( $P_{prod}$ ) is difficult due to its non-Gaussian distribution. Instead, the authors model the rotational speed ( $\Omega$ ) of the WEC's internal pendulum, which follows a more Gaussian distribution.
- They employ an Autoregressive (AR) model of order 2 (AR(2)) to capture the dynamics of $\Omega$ .
- Model Fitting: Standard Conditional Maximum Likelihood Estimation (CMLE) was found inadequate for reproducing the autocorrelation function (ACF) of the data at short time lags. Consequently, the authors implemented a "multi-lags ACF fitting" method, minimizing the squared difference between the theoretical and empirical ACFs over a 15-second horizon.
- The resulting AR(2) model is reformulated into a state-space representation using speed ( $\Omega$ ) and acceleration ( $A$ ) as state variables, alongside the stored energy ( $E_{sto}$ ). This yields a 3-dimensional Markovian state vector $x = (E_{sto}, \Omega, A)$ .
The StoDynProg Library:
- To solve the resulting optimization problem, the authors developed StoDynProg, a generic Python library for Stochastic Dynamic Programming (SDP).
- Architecture: The library uses SysDescription to define system dynamics and cost functions, and DPSolver to manage the discretization of the state space and execute algorithms.
- Algorithm: The library implements Policy Iteration, which is shown to be significantly more efficient than Value Iteration for this problem. Policy Iteration alternates between policy evaluation (solving linear equations for a fixed policy) and policy improvement (finding the optimal control for the current value function).
- Interpolation: A critical component is the use of a custom MultilinearInterpolator (adapted from the Dolo project) to handle n-dimensional interpolation on rectangular grids, a requirement for evaluating value functions between discrete grid points.
Optimization Execution:
- The state space is discretized into a grid of approximately 110,000 points ( $30 \times 60 \times 60$ for energy, speed, and acceleration, respectively).
- The algorithm is initialized with a simple linear heuristic control law and iterates until convergence (achieved in ~5 policy iterations).

Key Results

Convergence and Efficiency: The Policy Iteration algorithm converged in approximately 3 minutes on a standard desktop computer. In contrast, a Value Iteration approach would have required roughly 5 hours (100 times slower) due to the cost of optimization at every step.
Control Policy: The resulting optimal policy $\mu(E_{sto}, \Omega, A)$ is a state-feedback function. It confirms that higher stored energy leads to higher grid injection, but uniquely incorporates speed and acceleration to modulate power, effectively anticipating mechanical energy changes.
Performance Improvement: When applied to 1000-second simulation samples of the SEAREV:
- The optimized policy reduced the standard deviation of the injected power by approximately 20% compared to the linear heuristic policy (specifically 27%, 16%, and 22% across three samples).
- The optimized policy utilized the storage capacity more effectively, particularly at higher energy levels, smoothing "peaks and valleys" better than the linear approach.
Limitations: The authors note that in high-production scenarios with abrupt accelerations (exceeding 4 standard deviations), the linear AR model fails to capture the non-linear dynamics of the WEC. Consequently, the optimized policy produces "spiky" output in these specific regimes, performing slightly worse than the linear heuristic in those isolated instances.

Significance and Claims
The paper claims to demonstrate the practical application of Stochastic Dynamic Programming to a realistic energy management problem using a generic, open-source Python tool.

Tooling: The primary contribution is StoDynProg, a library designed to handle arbitrary dimensions of state and control spaces, addressing the "Curse of Dimensionality" by enforcing low-order Markovian models (limiting state dimensions to 3 or 4).
Methodological Rigor: The work highlights the importance of appropriate model fitting (multi-lags ACF vs. CMLE) for ensuring the stochastic model accurately reflects the system's correlation structure, which is critical for the DP solution.
Applicability: The authors assert that while the specific example is wave energy smoothing, the library and methodology are generic enough to be applied to other Optimal Control problems in electrical engineering, mechanical engineering, and life sciences, provided the system can be modeled as a Markovian process with a manageable state dimension.

The paper concludes modestly, noting that while the 20% reduction in variability is significant, the economic value of this improvement depends on grid codes and specific penalty criteria (e.g., "flicker") which are subjects for future research.