⚡ electrical engineering

Sizing of Battery Considering Renewable Energy Bidding Strategy with Reinforcement Learning

This paper proposes a novel, computationally efficient algorithm that co-optimizes Battery Energy Storage System (BESS) sizing and renewable energy bidding strategies by integrating Deep Recurrent Q-Networks with a distributed reinforcement learning framework inspired by embodied cognition to effectively manage uncertainties and enable parallel computation.

Original authors: Taiyo Mantani, Hikaru Hoshino, Tomonari Kanazawa, Eiko Furutani

Published 2026-02-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Taiyo Mantani, Hikaru Hoshino, Tomonari Kanazawa, Eiko Furutani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a farmer who grows apples (Renewable Energy). You have a big barn (the Battery) where you can store apples, and you sell your apples at a bustling weekly market (the Electricity Market).

Here is the problem:

The Weather is Unpredictable: Sometimes you get a huge harvest, sometimes very little.
The Market Prices Change: Sometimes apples are worth a fortune; other times, they are almost free.
The Penalty: If you promise to sell 100 apples but only have 80, you get fined. If you have 100 but only promised 50, you lose money on the extra 50.

The Old Way (The "Two-Stage" Method)
Traditionally, farmers tried to solve this in two separate steps:

Step 1: Buy a barn. You guess, "Maybe a barn that holds 500 apples is best."
Step 2: Hire a manager to decide when to sell or store apples based on that specific barn size.

The problem? If you guessed the wrong barn size, you have to fire the manager, buy a new barn, hire a new manager, and start all over again. It's slow, expensive, and you might never find the perfect combination.

The New Way (This Paper's Solution)
The researchers from the University of Hyogo propose a smarter approach using Reinforcement Learning (RL). Think of this as training a super-smart AI manager who learns by trial and error, like a video game character.

But here is the twist: They let the AI change the size of the barn while it is learning how to manage the apples.

The Creative Analogy: The "Shape-Shifting" Video Game

Imagine a video game where you are playing a character who needs to navigate a maze (the market) to collect coins (profit).

The Old Way: You pick a backpack size (Small, Medium, Large) before you start. You play the game 100 times with the Small backpack, then 100 times with the Medium, and so on. You compare the scores to see which backpack was best.
The New Way: You play the game, but every time you die and restart, the game randomly gives you a slightly different backpack size. As you play more, the game learns: "Hey, when the maze is tricky and prices are low, a Medium backpack works best. When it's easy, a Small one saves money."

The AI isn't just learning how to play; it's also learning what gear to wear to play best.

How It Works (The Magic Ingredients)

The "Brain" (DRQN): The AI uses a special type of memory (called a Deep Recurrent Q-Network) that remembers the past. It knows, "Last week the price was low, so I should have saved apples." It handles the uncertainty of the weather and the market better than old math formulas.
The "Co-Optimization": Instead of fixing the battery size (the barn) first, the algorithm treats the size as a variable. It constantly tweaks the size up and down during training.
- If a bigger battery helps make more money, the AI "learns" to prefer bigger batteries.
- If a smaller battery saves enough on construction costs to be more profitable, it "learns" to prefer smaller ones.
Parallel Play: To make this fast, they used 12 different "workers" (computers) playing the game at the same time. This is like having 12 farmers testing different barn sizes simultaneously, rather than one farmer doing it one by one.

Why This Matters

Speed: The old way of testing every possible barn size one by one is like trying to find a needle in a haystack by checking every single straw. This new method finds the needle much faster.
Safety: The researchers found that when they let the AI adjust the barn size while learning, the AI was less likely to "fail" (get stuck in a bad strategy). It was like giving the player a safety net; if the current strategy isn't working, changing the equipment (battery size) helps them recover.
Real-World Money: For renewable energy companies, this means they can figure out exactly how big of a battery to buy to make the most profit, without wasting money on a battery that is too big or too small.

The Bottom Line

This paper introduces a smart, efficient way to design renewable energy systems. Instead of guessing the size of the battery and then figuring out how to use it, the new method lets a computer learn both the best size and the best usage strategy at the same time. It's like teaching a robot to build its own perfect toolbox while learning how to fix a car, ensuring it ends up with the perfect setup for the job.

1. Problem Statement

The integration of Variable Renewable Energy (VRE) sources, such as solar and wind, into electricity markets creates significant challenges due to generation uncertainty and price volatility. Renewable producers must participate in real-time markets to maximize profits while minimizing penalties for dispatch deviations.

The Core Challenge: Determining the optimal size of a Battery Energy Storage System (BESS) is critical for operational flexibility (arbitrage and imbalance mitigation). However, the optimal BESS size is intrinsically linked to the bidding strategy.
Limitations of Existing Methods:
- Two-Stage Approaches: Most existing methods separate the problem into two stages: first training a bidding policy for a fixed battery size, then optimizing the size. This requires training models across a vast search space, leading to prohibitive computational costs.
- Simplifications: Many current sizing methods use rule-of-thumb models that ignore complex market dynamics and price uncertainties.
- Computational Burden: Mathematical optimization techniques (e.g., stochastic programming) require simulating numerous scenarios, while robust optimization can be overly conservative.

2. Methodology

The authors propose a computationally efficient co-optimization algorithm that simultaneously optimizes the BESS size and the bidding policy using an extended Reinforcement Learning (RL) framework.

A. Market and System Model

Market Structure: A generic real-time market where producers bid ( $b_t$ ) one time step ahead. Deviations between the bid and actual dispatch ( $x^D_t$ ) incur imbalance penalties based on surplus and deficit prices.
Objective: Maximize total net profit ( $F$ ), defined as market revenue minus imbalance penalties and battery degradation costs.
Constraints: The model accounts for State of Charge (SOC) limits, charging/discharging efficiencies, and power limits proportional to the battery capacity ( $E_{max}$ ).

B. Reinforcement Learning Framework

Algorithm Base: The system uses a Deep Recurrent Q-Network (DRQN) combined with a distributed RL framework (inspired by Ape-X).
- Why DRQN? It integrates Long Short-Term Memory (LSTM) to handle Partially Observable Markov Decision Processes (POMDPs), allowing the agent to learn from sequences of observations (past generation, prices, and SOC) rather than just instantaneous states.
Action Space: The agent determines the bid ( $b_t$ ) and a scaling factor ( $\rho_t$ ) for energy arbitrage. Charging and discharging powers are derived from the difference between generation and bid, scaled by $\rho_t$ .
Reward Function: Designed to maximize net profit while reducing variance in learning, formulated as the net profit minus the baseline revenue from selling generation directly.

C. The Co-Optimization Extension

The novel contribution is the extension of the standard RL loop to include the BESS size ( $E_{max}$ ) as a learnable parameter:

Augmented Q-Function: The Q-function is extended to $Q(o_t, h_t, c_t, a_t, \omega)$ , where $\omega = \{E_{max}\}$ is the design parameter.
Policy for Design: Instead of fixing $\omega$ , a distribution $p_\mu(\omega)$ (Gaussian) is maintained. The mean $\mu$ is a learnable parameter representing the "belief" about the optimal battery size.
Training Loop:
- At the start of each episode, a specific battery size $\omega$ is sampled from $p_\mu$ .
- The DRQN weights ( $\theta$ ) are updated based on the trajectory generated with that specific size.
- Every $N_{up}$ episodes, the mean $\mu$ is updated using a policy gradient theorem based on the total return (profit minus annualized battery installation cost).
Parallelization: The framework utilizes multiple workers to collect data in parallel, handling long-term data efficiently.

3. Key Contributions

Novel Co-Optimization Algorithm: This is the first application of a co-optimization algorithm (inspired by embodied cognition) to energy systems, simultaneously optimizing the physical design (battery size) and the control policy (bidding strategy).
Computational Efficiency: By avoiding the exhaustive search required in two-stage methods, the proposed method significantly reduces computational time while finding global or near-global optima.
Robustness to Training Failure: The study demonstrates that co-optimizing the system configuration helps avoid training failures (negative rewards) that often occur when the fixed system design is suboptimal for the learned policy.
Integration of DRQN and Distributed RL: The combination of DRQN (for temporal dependencies) and distributed sampling (for long-term data handling) effectively manages the uncertainties of renewable generation and market prices.

4. Experimental Results

The authors validated the method using both short-term (1 week) and long-term (1 year) data from the Japanese electricity market (JEPX) and solar generation data (METPV-20).

Short-Term Verification (1 Week):
- Accuracy: The co-optimization algorithm successfully converged to optimal battery sizes (e.g., ~120% capacity for low installation costs, ~50% for medium costs) that matched the results of an exhaustive two-stage search.
- Efficiency: While the co-optimization took longer to converge than a fixed-size training (48 mins vs. 8 mins), it was significantly faster than running the exhaustive search required to find the optimal size in the two-stage approach.
Long-Term Applicability (1 Year):
- Training Stability: When using a full year of data, training with a fixed battery size failed in 87.5% of experiments (resulting in negative rewards). In contrast, the co-optimization method reduced the failure rate to 37.5%.
- Conclusion: Dynamically adjusting the battery size during training allows the agent to explore better system configurations, preventing the policy from getting stuck in suboptimal or failing states.

5. Significance

This paper addresses a critical gap in the operational planning of renewable energy systems. By treating the physical infrastructure (battery sizing) and the operational strategy (bidding) as a coupled problem solvable via a single learning loop, the authors provide a scalable solution for:

Business Model Development: Enabling renewable producers to determine the most profitable investment size for storage without prohibitive computational costs.
Market Integration: Improving the ability of VRE sources to participate in real-time markets by reducing imbalance penalties and maximizing arbitrage opportunities.
Future Research: The work opens a new direction for applying "embodied cognition" concepts to power systems, suggesting that hardware design and control software should be optimized concurrently rather than sequentially.