⚡ electrical engineering

Learning Reachability of Energy Storage Arbitrage

This paper proposes an end-to-end learning framework that integrates a stopping-time reward and a state-of-charge range target penalty into online optimization to align energy storage arbitrage incentives with system reliability, thereby enhancing the reachability of critical reserve levels while improving profitability and stability under volatile market conditions.

Original authors: Tomás Tapia, Agustin Castellano, Enrique Mallada, Yury Dvorkin

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tomás Tapia, Agustin Castellano, Enrique Mallada, Yury Dvorkin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a battery as a smart savings account for electricity. Its job is simple: buy energy when it's cheap (like stocking up on groceries on sale) and sell it when it's expensive (like selling those groceries at a markup). This is called "arbitrage."

However, there's a catch. The power grid is like a busy highway that sometimes gets jammed by sudden storms or heatwaves. When these "critical hours" hit, the grid desperately needs batteries to have a full tank of energy ready to go. If a battery sells all its energy too early just to make a quick buck, it might leave the grid stranded exactly when it needs help most.

This paper tackles a specific problem: How do we teach a battery to be greedy for profit but also responsible for the grid's safety?

Here is the breakdown of their solution using everyday analogies:

1. The Problem: The "Myopic" Battery

Without special instructions, a battery acts like a short-sighted shopper. It sees a high price spike and immediately sells everything it has. It doesn't care that a bigger storm is coming in 5 hours that will require it to have a full tank. It makes money now but fails the system later.

2. The Solution: The "Stop-and-Keep" Signal

The authors introduce a new rule called a "Stopping-Time Reward."

Think of this like a traffic light for the battery's decision-making process:

Green Light (Arbitrage Mode): The battery is free to buy and sell energy to make money.
Red Light (Preservation Mode): At a specific time (the "stopping time"), the battery gets a bonus reward if it simply stops trading and holds onto its energy.

This reward acts as a gentle nudge. It tells the battery: "If you stop selling now and keep your tank full, we will pay you a bonus. This ensures you are ready for the critical hours later."

3. The Three "Coaches" (Models) Tested

The paper tests three different ways to teach the battery this behavior, comparing them like three different coaches training an athlete:

Coach A (SAA - The "Historian"): This coach looks at thousands of past weather and price scenarios to calculate the perfect plan.
- Pros: It's very accurate if the future looks like the past.
- Cons: It's slow and computationally heavy, like trying to solve a massive puzzle before every single move.
Coach B (DQN - The "Gambler"): This coach uses a "trial and error" approach (Reinforcement Learning). It learns by playing the game over and over, getting punished if it runs out of energy and rewarded if it makes money.
- Pros: It's fast and adapts well to new situations.
- Cons: It can be unpredictable. Sometimes it gets too greedy, sometimes it panics. The paper found it had high "variance," meaning its performance was a wild rollercoaster—sometimes making huge profits, sometimes failing badly.
Coach C (E2E - The "Smart Strategist"): This is the paper's main innovation. It uses an End-to-End (E2E) learning framework.
- How it works: Instead of just guessing prices, this coach learns to predict prices specifically to help the battery make the best decisions. It connects the "prediction" brain directly to the "decision" brain.
- The Result: It's like a coach who doesn't just tell the athlete what to do, but teaches them how to interpret the weather forecast to make the right move.
- Performance: The E2E model was the most stable. It didn't make the highest possible profit in every single scenario, but it avoided the disastrous failures. It consistently kept the battery's "tank" full enough for the critical hours while still making good money.

4. The "Point of No Return"

The paper also talks about a concept called "Reachability."
Imagine you are driving to a destination (the "Critical Hour"). There is a point on the road where, even if you drive at maximum speed, you can no longer make it on time.

The paper's method helps the battery identify this "Point of No Return" early.
Once the battery realizes it's getting close to this point, the "Stopping-Time Reward" kicks in, forcing it to switch from "racing for profit" to "driving safely to the destination."

Summary of Results

Reliability: The new method (E2E) successfully ensured the battery had enough energy left for the critical hours (the "target range") much more consistently than the other methods.
Stability: While the "Gambler" (DQN) model made big swings in profit, the "Smart Strategist" (E2E) kept profits steady and reduced the risk of the battery running empty.
The Trade-off: By forcing the battery to be more reliable, it sometimes made slightly less "quick cash" than a purely greedy battery, but it prevented the system from failing when it mattered most.

In a nutshell: The paper proposes a new way to pay batteries. Instead of just paying them for every kilowatt they sell, we pay them a bonus for stopping early and holding their energy when a storm is coming. This aligns the battery's desire for profit with the grid's need for safety, and the new "Smart Strategist" AI is the best at figuring out exactly when to hit the brakes.

Technical Summary: Learning Reachability of Energy Storage Arbitrage

Problem Statement
Power systems are increasingly subject to weather-driven variability, necessitating flexible, energy-limited storage resources to buffer fluctuations. However, the economic value of energy storage is time-dependent and relies on intertemporal decisions under uncertain prices. Without accounting for the future reliability value of stored energy, battery assets may act myopically, discharging too early or failing to preserve reserves during "critical hours" (periods of high system stress). This behavior leads to suboptimal economic performance and reduced system reliability.

Two primary challenges arise when attempting to internalize reliability into market signals for individual storage assets:

Reachability: Determining if a sequence of control actions exists that allows the battery to reach and maintain a desired State-of-Charge (SoC) range by a critical time $\tau$ , satisfying operational limits for a minimum fraction of uncertainty realizations.
Probability Quantification: Calculating the probability that the terminal SoC will lie within a target range during critical hours. Analytical solutions to these chance-constrained problems are often computationally expensive or intractable under realistic, heavy-tailed price dynamics.

Existing mechanisms, such as Minimum State of Charge (MSOC) requirements, often introduce efficiency losses or implementation complexity. Furthermore, standard machine learning approaches for inventory control do not directly internalize the "stopping-time" decision—the specific moment to transition from arbitrage to reserve preservation.

Methodology
The paper proposes a framework that integrates a stopping-time reward with a State-of-Charge (SoC) range target penalty to align arbitrage incentives with system reliability. The core objective is to reward storage assets that transition into and maintain a reserve-preserving mode sufficiently early to ensure a feasible trajectory to a target SoC before critical hours.

The authors formulate the problem as an online optimization with a chance-constrained terminal SoC and evaluate three distinct approaches:

Sample Averaged Approximation (SAA): A benchmark formulation that replaces the uncertainty representation with empirical averages using historical scenarios. It employs a mixed-integer reformulation to enforce the chance constraint, allowing for the identification of optimal stopping times but suffering from high computational complexity as the number of scenarios increases.
Deep Q-Learning (DQN): A model-free reinforcement learning approach that frames the arbitrage problem as a Markov Decision Process (MDP). The state space includes discretized SoC, a stopping indicator, and price bins. The reward function includes arbitrage profits, a stopping-time reward (activated upon entering reserve mode), and a terminal penalty for violating the SoC target band. While scalable to high-dimensional observations, this method can exhibit high variance and sensitivity to penalty tuning.
End-to-End (E2E) Learning Framework: The proposed primary contribution. This approach jointly trains a price predictor (represented as an uncertainty set) and a downstream differentiable optimization task.
- Uncertainty Quantification: The predictor outputs a center price and an uncertainty set calibrated using conformal prediction to achieve finite-sample coverage for future price realizations.
- Robust Optimization: The dispatch layer solves a robust arbitrage problem under the worst-case scenario within the calibrated uncertainty set.
- Task-Driven Training: Instead of minimizing generic forecast errors, the predictor is optimized to maximize the downstream objective (arbitrage profit + stopping-time reward). Gradients are back-propagated through the optimization layer to the predictor, aligning forecasts with operational reliability goals.

Key Contributions

Stopping-Time Reward Integration: The paper extends the battery arbitrage problem by introducing a stopping-time reward and terminal SoC penalty. This internalizes the operator's preference to preserve energy before critical hours, steering assets toward feasible SoC trajectories and improving reachability.
E2E Framework for Reliability: The authors embed the stopping-time reward within an E2E learning framework. This approach calibrates the energy-price predictor to a specific downstream optimization task, providing robustness guarantees. It aims to increase profit under worst-case price realizations while maintaining a user-defined likelihood of meeting the terminal SoC target.
Empirical Comparison: The study provides a comparative analysis of the E2E approach against SAA and DQN baselines using real day-ahead price data from PJM. The results highlight the trade-off between reliability provision and asset profit, demonstrating that time-indexed control signals (stopping-time rewards) can steer batteries toward reliability-consistent behavior.

Results
Using six years of PJM data with a 24-hour horizon and a critical hour at $t=24$ , the models were evaluated across varying stopping-reward levels ( $c$ ):

SAA: Achieved higher profits due to the lack of penalty terms in the baseline but showed stepwise shifts in stopping times as rewards increased.
DQN: Produced the highest profit magnitudes in certain reward ranges (e.g., $c=15$ ) but exhibited high variance (up to 128% standard deviation) and occasional feasibility violations. The learned stopping times were sensitive to penalty scaling.
E2E: Delivered lower variability in profit (standard deviation reduced to 1-3%) and more consistent stopping-time decisions. While the expected profit was lower than DQN in some low-reward scenarios due to its robust design, it maintained high marginal coverage (above 94%) across price realizations. As the stopping reward increased, the E2E model's profit approached that of other models while maintaining stability.

Significance and Claims
The paper claims that the proposed design enhances the reachability of target SoC ranges and improves profit stability under volatile conditions. By introducing a stopping-time reward, the framework acts as a time barrier that triggers before the "point of no return," switching the policy from arbitrage to preservation mode. This aligns private arbitrage incentives with the systemic public objective of resilience without requiring administrative rules outside of market clearing.

The authors assert that the E2E approach offers a market-compatible alternative to rigid minimum SoC requirements. It reconciles system and asset perspectives by endogenizing the value and likelihood of terminal SoC reachability. The study concludes that the E2E model yields the least variability in decisions and stable stopping-times, whereas DQN, while potentially more profitable, suffers from higher variance and feasibility risks. Future work is suggested to extend this framework to predictive control and further quantify system operator risk through time-series dynamics.