Multi-market value-stacking: Battery control for combined imbalance participation and non-uniform FCR bidding

Technical Summary: Multi-market Value-stacking with Non-uniform FCR Bidding

Problem Statement
The increasing penetration of Renewable Energy Sources (RES) in European power systems has heightened frequency deviations and system imbalances, necessitating robust ancillary services like Frequency Containment Reserve (FCR). Battery Energy Storage Systems (BESS) are ideal for these services due to their rapid response; however, they are classified as Limited Energy Resources (LERs). This classification requires strict management of the State of Energy (SoE) to ensure continuous availability for both upward and downward FCR activation.

Existing research typically employs uniform FCR bidding, where reserve capacity commitments remain static throughout the control period. This static approach fails to fully exploit BESS flexibility because it does not dynamically balance the trade-off between reserving energy for FCR delivery and utilizing that energy for imbalance arbitrage. While Deep Reinforcement Learning (DRL) has been applied to real-time imbalance control, it has largely operated under these static, uniform bid constraints. Conversely, non-uniform bidding (time-varying commitments) has shown promise in deterministic settings but has not been effectively integrated with high-resolution, stochastic real-time control. This paper addresses the gap: can additional revenue be unlocked by combining non-uniform FCR bids with real-time imbalance control?

Methodology
The authors propose a two-stage control framework designed for the European context to decouple long-horizon bidding decisions from fast real-time control:

1. Stage 1: Data-Driven Non-Uniform FCR Bid Optimization

   • Objective: Derive a time-varying sequence of FCR bids (in 4-hour blocks) that maximizes expected profit without relying on perfect foresight for the real-time layer.
   • Approach: A data-driven Monte Carlo (MC) optimization is performed ex-post using realized frequency and price data. For each 4-hour block, the system evaluates discrete candidate bids (0 to $P_{nom}-1$ MW).
   • Simulation: Each candidate bid is tested via 50 MC simulations at 1-second resolution, incorporating historical FCR activations and financial settlements.
   • Heuristic Control: A simple rule-based heuristic manages SoE during these simulations to ensure FCR constraints are met while capturing imbalance revenues.
   • Adjustment: To prevent bias from end-of-block SoE levels, a terminal-value adjustment is applied to the profit metric, accounting for the opportunity cost of the remaining energy in the subsequent block. The bid with the highest expected adjusted profit is selected for the block.

2. Stage 2: DRL-Based Real-Time Imbalance Control

   • Objective: Exploit the residual flexibility (the difference between nominal power and the Stage 1 bid) for real-time imbalance trading while maintaining SoE feasibility.
   • Agent: A Double Deep Q-Network (DDQN) operates at 1-minute intervals, aligned with imbalance price signal updates.
   • State Space: The agent observes market signals (imbalance prices, time features), physical state (SoE, cumulative cycle usage), and constraint context (distance to SoE bounds, current/upcoming FCR commitments).
   • Action Space: Discrete charge, idle, or discharge decisions mapped to the available residual flexibility.
   • Safety Mechanism: A two-tier safety layer ensures compliance: (i) an action mask filters out setpoints likely to violate constraints, and (ii) a reactive override corrects actions if stochastic FCR activations push the SoE outside bounds.
   • Reward Function: A multi-objective function balances net imbalance revenue, operational safety (penalizing SoE proximity/violation), and battery health (penalizing excessive daily cycle throughput).

Key Contributions

• Novel Framework: Introduction of a two-stage architecture that integrates non-uniform FCR bidding with DRL-based real-time control, addressing the limitation of static bids in existing literature.
• Proof of Concept: Demonstration that time-varying bidding strategies can be derived via data-driven MC optimization to create a dynamic "feasibility envelope" for the real-time controller.
• Algorithmic Implementation: Application of DDQN to manage the complex trade-off between FCR compliance and imbalance arbitrage under time-varying constraints, utilizing a specific reward structure to manage battery degradation.

Results
The framework was evaluated using a 10 MW / 20 MWh BESS in the Belgian market over the full year of 2022, using high-resolution frequency and market data.

• Profit Increase: The proposed non-uniform strategy achieved a total net profit of €3.06 million, representing a 7.56% increase over the best-performing uniform baseline (€2.84 million).
• Revenue Composition: The FCR-specific revenue remained comparable between the non-uniform and best uniform strategies, indicating the optimizer effectively allocated capacity during peak FCR price periods. The primary gain (approx. 7.5%) stemmed from increased imbalance arbitrage.
• Operational Dynamics: By dynamically reducing FCR obligations during periods of high imbalance volatility, the non-uniform strategy expanded the operational SoE margins. This allowed the DRL agent to execute more aggressive arbitrage strategies that would be restricted under static bids.
• Asset Health: Despite a higher number of charging cycles compared to uniform baselines, the agent successfully kept total throughput within the prescribed daily cycle limit ($C_{max}$), demonstrating effective utilization of available headroom without violating longevity constraints.
• Bid Behavior: The optimizer selected larger FCR bids when FCR prices were high and reduced bids when imbalance price volatility increased, creating a diagonal transition pattern between FCR capacity and imbalance flexibility.

Significance and Claims
The paper claims that non-uniform bidding unlocks additional value by more effectively aligning reserve obligations with rapidly changing imbalance opportunities. The study posits that the "bid then control residual flexibility" architecture provides a reusable abstraction for other value-stacking scenarios involving time-dependent constraints (e.g., aFRR, hybrid PV-BESS).

The authors maintain a modest stance regarding the current implementation:

• The Stage 1 optimization relies on ex-post data, serving as a proof of concept and an upper-bound benchmark rather than a deployable forecasting tool.
• The use of DDQN is presented as a stable baseline to isolate the impact of the bidding strategy rather than to benchmark RL algorithms against state-of-the-art alternatives.
• Future work is identified as necessary to integrate proactive, forecasting-based bid selection and to explore more robust RL architectures.

The results suggest that moving beyond static, uniform commitments allows BESS operators to capture significant additional revenue in multi-market environments while maintaining strict compliance with ancillary service requirements.