Optimal design of solar-battery hybrid resources considering multi-market participation under weather and price uncertainty

Technical Summary: Optimal Design of Solar-Battery Hybrid Resources Considering Multi-Market Participation Under Weather and Price Uncertainty

Problem Statement
The rapid integration of variable renewable energy (VRE) necessitates flexible, coordinated resources capable of participating in multiple electricity markets simultaneously. While hybrid resources—integrating renewable generation and battery storage within a single market entity—offer significant operational and financial advantages over co-located (separately bid) resources, their optimal design remains a complex challenge. The core difficulty lies in the coupling of long-term investment decisions (system sizing) with short-term operational optimization under stochastic conditions (renewable generation and market price uncertainty).

Conventional optimization approaches often rely on deterministic forecasts or simplified uncertainty models, which may fail to capture the nonlinear dynamics and operational constraints inherent in multi-market participation (energy, ancillary services, and capacity markets). Furthermore, existing literature frequently treats system design and operational strategies as separate problems or focuses primarily on energy markets, neglecting the intricate trade-offs involved in ancillary service (AS) provision and the specific advantages of DC-coupled hybrid architectures, such as the recovery of clipped photovoltaic (PV) energy.

Methodology
To address these challenges, the authors propose a Deep Reinforcement Learning (DRL)-based co-optimization framework. This approach integrates system design variables directly into the policy learning process, enabling the joint optimization of hybrid system sizing and coordinated multi-market bidding strategies within a unified stochastic formulation.

• Framework Architecture: The method extends previous DRL-based co-optimization work by incorporating multi-market participation. The framework utilizes a Long Short-Term Memory (LSTM) network to handle partial observability of market and renewable variables, processing time-series data of PV generation and market prices. The agent employs a Deep Deterministic Policy Gradient (DDPG) algorithm to learn operational policies.
• Design Optimization: Unlike traditional bi-level approaches that separate design and operation, this framework treats the system design vector (PV capacity, battery energy capacity, and battery power rating) as a parameter of a stochastic policy. The distribution of design parameters is updated via policy gradient methods based on the cumulative economic returns (revenue minus costs) observed during training episodes.
• Constraint Handling: A critical component of the methodology is the "serial capacity allocation" strategy. To ensure feasibility under uncertainty, the agent's actions are transformed into feasible bids through a sequential allocation process:
  1. Contingency Reserves: Allocated first, primarily from battery discharge capability, respecting State-of-Charge (SOC) limits and duration requirements.
  2. Regulation Services: Upward and downward regulation capacities are allocated next, accounting for remaining battery margins and a reliability factor for PV generation forecasts.
  3. Energy Market Bids: The remaining feasible operating range determines the forward energy market bid, explicitly accounting for potential imbalances.
• Market Model: The study models participation in forward energy markets, contingency reserves, regulation services, and capacity markets. It explicitly accounts for PV forecast errors, modeling energy imbalances as penalty costs rather than relying on real-time arbitrage assumptions. The framework distinguishes between DC-coupled hybrid resources (single entity, shared inverter) and AC-coupled co-located resources (separate entities).

Key Contributions
The paper identifies and addresses three specific research gaps:

1. Constraint-Aware DRL for Hybrid AS Provision: While DRL has been applied to battery storage, this work extends the framework to PV-battery hybrids with hard operational constraints. It enables joint AS participation of both PV and battery under forecast uncertainty, moving beyond soft constraints or perfect forecast assumptions found in prior studies.
2. Quantitative Comparison of Market Structures: The paper provides a systematic quantitative comparison between hybrid and co-located participation models under multi-market settings. It evaluates differences in operational flexibility, bidding strategies, and economic performance, offering insights into how market structures influence resource value.
3. Uncertainty-Aware Co-Optimization of Design and Operation: The primary contribution is the integration of multi-market participation into the design optimization of PV-battery systems under uncertainty. Unlike previous works that rely on deterministic forecasts for sizing, this framework learns optimal capacities and operational policies simultaneously, directly optimizing long-term economic performance in stochastic environments.

Results
Numerical case studies using historical CAISO data demonstrate the framework's effectiveness:

• Hybrid vs. Co-located: Under identical system sizes, hybrid resources consistently achieved higher rewards than co-located resources. The hybrid configuration reduced imbalance penalties by approximately 40% (from $3,622 to $2,193 in a sample week) by internally mitigating PV forecast errors. Additionally, the DC-coupled architecture allowed for the recovery of "clipped" PV energy, shifting it to high-price periods, thereby increasing energy market revenue.
• Co-Optimization Outcomes: The joint optimization of design and operation yielded configurations that outperformed fixed-design baselines. For instance, in a baseline scenario, the framework identified an optimal battery duration of 2.71 hours, which was below the 4-hour capacity market requirement, yet remained profitable due to energy and AS market revenues.
• Policy Sensitivity: The framework successfully adapted to hypothetical policy changes. When the capacity market duration requirement was increased to 8 hours, the optimal design shifted toward larger PV capacity and smaller battery capacity, indicating that duration mandates alone do not guarantee battery deployment if they conflict with economic viability. Conversely, increased AS prices for upward regulation drove the optimal design toward larger battery power ratings.
• Long-Term Applicability: Using a distributed RL framework with one year of data, the study confirmed that seasonal variations significantly influence revenue structures and optimal strategies. The distributed approach reduced training time for full-year optimization to approximately 10 hours, demonstrating computational tractability for long-term planning.

Significance and Claims
The paper claims that its proposed framework offers a robust tool for identifying economically rational hybrid system configurations that are resilient to weather and price uncertainties. By embedding design variables into the learning process, the method avoids the computational burden of bi-level optimization and captures the complex interplay between system sizing and multi-market operational strategies.

The authors emphasize that their work provides new insights into the value of hybrid resources, particularly their ability to recover clipped energy and mitigate imbalances, which are often overlooked in studies focusing solely on energy markets or assuming perfect forecasts. The study concludes that while hybrid resources offer superior flexibility and profitability, their optimal design is highly sensitive to market rules (such as capacity duration requirements) and price signals, necessitating adaptive, uncertainty-aware planning tools. The authors modestly note that future work could extend this framework to settings involving multiple decision-makers, such as virtual power plants.