Technical Summary: MARS-DA

Problem Statement

The increasing penetration of renewable energy has introduced significant volatility into wholesale electricity markets, complicating optimal bidding strategies for power producers. Traditional model-based approaches (stochastic or robust optimization) struggle with scalability and adaptability in dynamic strategic environments. Conversely, standard Deep Reinforcement Learning (DRL) methods often employ flat structures and fixed policies that are prone to overfitting and fail to manage the trade-off between profit maximization and risk management, particularly during market regime shifts.

A critical challenge identified is the tension between securing conservative Day-Ahead (DA) commitments for risk control and exploiting opportunistic Real-Time (RT) arbitrage to capture volatility. Existing methods lack the hierarchical, risk-aware structures necessary to adaptively coordinate these conflicting objectives under uncertainty.

Methodology: The MARS-DA Framework

The authors propose MARS-DA (Multi-Agent Regime-Switching for Day-Ahead markets), a hierarchical reinforcement learning framework designed to optimize bidding in two-settlement electricity markets. The framework decouples strategy learning from regime adaptation using a "Manager-Worker" architecture trained via Proximal Policy Optimization (PPO).

1. Hierarchical Architecture

• Base Agents (Workers): Two specialized agents are pre-trained in a "University Phase" to master distinct market roles:
  • Safe Agent: Optimizes for reliable DA allocation. Its reward function scales profit by the DA allocation ratio and heavily penalizes RT market utilization, enforcing a conservative, risk-averse posture.
  • Speculator Agent: Optimizes for volatile RT arbitrage. Its reward function scales profit by the RT allocation ratio and penalizes DA locking, encouraging aggressive strategies to capture price spreads.
• Meta-Controller (Manager): A top-level policy network that observes the market state and dynamically generates blending weights ($w_{safe}, w_{spec}$) to combine the actions of the frozen base agents. This allows the system to transition seamlessly between conservative and aggressive postures without retraining the underlying policies.

2. Risk-Aware Objective

The Meta-Controller is trained to maximize a risk-adjusted return rather than raw profit. The reward function employs a concave utility shaping:
$$r_{meta} = \frac{\pi_t}{S_{linear}} - \lambda_{risk} \cdot \frac{1}{2} \left( \frac{\pi_t}{S_{var}} \right)^2$$
This formulation penalizes the squared magnitude of realized profit, discouraging "jackpot-seeking" strategies that carry catastrophic downside risk and encouraging consistent, reliable gains.

3. High-Fidelity Simulation Environment

The paper introduces StrategicBiddingEnv, an open-source, Gymnasium-compatible environment grounded in empirical data from the PJM Interconnection (2018–2025).

• Data: Integrates DA and RT Locational Marginal Prices (LMPs), load forecasts, weather data, and natural gas prices.
• Mechanics: Explicitly models the two-settlement structure where uncommitted DA capacity ($q_{RT} = P_{max} - q_{DA}$) is automatically settled in the RT market. It accounts for marginal fuel costs, startup costs, and operational penalties (ramping limits, minimum up/down times).

Key Contributions

1. Open-Source Environment: Development of a high-fidelity simulation environment that captures the two-settlement market structure and the coupling between DA commitments and RT deviations, providing a standardized testbed for general and risk-sensitive agents.
2. Hierarchical Framework: Proposal of MARS-DA, a novel framework utilizing adaptive risk-aware policy blending. It orchestrates heterogeneous agents (Safe vs. Speculator) via a Meta-Controller to navigate market uncertainty.
3. Empirical Validation: Extensive experiments demonstrating that MARS-DA achieves superior risk-adjusted performance and stability compared to state-of-the-art baselines, particularly during periods of extreme market volatility.

Experimental Results

The framework was evaluated on PJM data across two test periods: 2022 (high fuel prices, post-pandemic recovery) and 2024–2025 (high volatility, increased renewable penetration).

• Performance Metrics: MARS-DA outperformed baselines including Vanilla PPO, Soft Actor-Critic (SAC), CVaR-PPO, and a Rolling Horizon MPC heuristic.
  • Test Period 1 (2022): Achieved the highest Sharpe Ratio (0.978) and lowest Maximum Drawdown (2.84%).
  • Test Period 2 (2024–2025): Despite significantly higher volatility, MARS-DA maintained a Sharpe Ratio of 0.270 and a Maximum Drawdown of 8.12%, whereas baseline agents suffered drawdowns exceeding 55%.
• Regime Alignment: Analysis of the Meta-Controller revealed a "contrarian" hedging behavior. During extreme volatility spikes, the controller dynamically reduced the weight of the Speculator agent, acting as an automated "circuit breaker" to avoid catastrophic drawdowns.
• Ablation Studies:
  • Heterogeneity: The heterogeneous ensemble (Safe + Speculator) significantly outperformed a homogeneous baseline, confirming that functional role specialization drives performance.
  • Dynamic Weighting: Replacing the learned Meta-Controller with static 50/50 weights resulted in worse risk-adjusted returns (Sharpe 0.458 vs. 0.920), proving the necessity of state-dependent switching.
  • Ensemble Size: Adding a third "Neutral" agent degraded performance, suggesting the 2-agent configuration offers the optimal balance between diversity and training stability.

Significance and Claims

The paper claims that MARS-DA addresses the brittleness of monolithic RL policies and rigid optimization models by decoupling strategy learning from regime adaptation. The framework's primary significance lies in its ability to generalize to out-of-distribution stress tests, where it was the only agent to avoid catastrophic losses.

The authors assert that the Meta-Controller's learned "contrarian" hedging strategy—automatically reducing speculative exposure during extreme volatility—establishes a viable path toward autonomous, safe, and profitable participation in increasingly volatile, renewable-dominated power markets. The work emphasizes that risk management in electricity markets requires dynamic, hierarchical structures capable of balancing conservative commitments with opportunistic arbitrage.