🤖 AI

Towards Affordable Energy: A Gymnasium Environment for Electric Utility Demand-Response Programs

This paper introduces DR-Gym, an open-source, Gymnasium-compatible simulation environment designed to train and evaluate electric utilities in optimizing demand-response programs by modeling realistic market-level interactions, regime-switching wholesale prices, and physics-based building demands to enhance energy affordability and grid flexibility.

Original authors: Jose E. Aguilar Escamilla, Lingdong Zhou, Xiangqi Zhu, Huazheng Wang

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jose E. Aguilar Escamilla, Lingdong Zhou, Xiangqi Zhu, Huazheng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a giant, invisible water tank that supplies a whole neighborhood. Sometimes, the price of water from the main reservoir (the wholesale market) stays low and steady. But other times, during a heatwave or a freeze, the price suddenly skyrockets, and if you don't manage it, your customers' bills could become so high they can't pay them.

This paper introduces a new video game simulator called DR-Gym. It's designed to help the "water managers" (electric utilities) learn how to keep prices affordable for everyone, even when the market goes crazy.

Here is how the simulator works, broken down into simple parts:

1. The Problem: The "Wholesale Rollercoaster"

Think of the electricity market like a rollercoaster. Most of the time, it's a gentle ride. But during extreme weather (like a heatwave), the track suddenly goes vertical. Prices can jump from pennies to thousands of dollars in an hour.

The Risk: If a utility company just passes these crazy prices to customers, a family's bill could jump from $100 to $10,000 overnight.
The Goal: The utility wants to smooth out this ride. They want to pay customers a little bit of money to use less electricity during those scary high-price moments, which saves the utility money and protects the customers.

2. The Missing Piece: The "Training Gym"

Before, if you wanted to teach a computer (an AI) how to manage this, you had to use old, static data. It was like trying to learn to drive a car by looking at a map of a road that doesn't move.

The Flaw: Real life is interactive. If you offer a discount, people react. If you offer it too much, they get tired of it. Old simulators didn't capture this "conversation" between the utility and the customers.
The Solution: The authors built DR-Gym. It's a "Gymnasium" (a training environment) where an AI agent can practice making decisions in a world that reacts back. It's like a flight simulator for electric utilities.

3. How the Simulator Works (The Engine)

The simulator has three main parts that make it feel real:

The Price Engine (The Weather): It doesn't just pick random prices. It uses a "regime-switching" model. Imagine the weather forecast: usually sunny, but sometimes a storm hits and lasts for hours. This engine simulates those "price storms" that cluster together, just like real life, rather than just random, isolated spikes.
The House Engine (The Buildings): It uses real physics-based data from thousands of virtual homes. It knows that houses get hot in the afternoon and cold at night, and that appliances use power in specific patterns.
The People Engine (The Customers): This is the most unique part. The simulator knows that people are different.
- Some are Price-Sensitive (they jump at any discount).
- Some are Eco-Conscious (they help even for small reasons).
- Some are Reluctant (they need a big offer to move).
- The "Fatigue" Factor: Crucially, the simulator knows that if you ask the same people to cut power every single day, they get annoyed and stop listening. The AI has to learn when to ask and how much to offer so people don't get tired of it.

4. The Goal: Balancing the Scale

The AI agent in the gym has a tricky job. It has a daily budget of money to give out as credits. It has to decide:

Do I pay $0.05 to save $1.00?
Do I save my money for a bigger storm later?
Do I protect the customers from huge bills, or do I make sure the utility company still makes a profit?

The paper shows that the AI can learn to balance these goals. In their tests, the AI learned to be smarter than simple rules (like "always pay $0.05"). It learned to save its budget for the worst moments, protecting customers from the biggest financial shocks while still keeping the utility profitable.

5. Why This Matters

The authors say this is the first "standardized gym" specifically for electric utilities to practice these kinds of decisions.

For Researchers: It's a safe place to test new AI ideas without risking real people's money.
For Society: The ultimate goal is to design systems that protect vulnerable families from getting bankrupted by extreme weather, while keeping the lights on.

In short: The paper presents a sophisticated video game where an AI learns to be a smart, empathetic electricity manager, balancing the books and protecting customers from financial disaster during price storms.

Technical Summary: DR-Gym – A Gymnasium Environment for Electric Utility Demand-Response Programs

Problem Statement
Residential consumers face catastrophic financial risks due to extreme weather events and volatile wholesale electricity markets, exemplified by price spikes in CAISO and ERCOT (e.g., Winter Storm Uri). While Demand Response (DR) programs offer a mechanism to mitigate these risks by issuing financial credits during high-price periods to curtail load, optimizing the sequential decision-making process for credit issuance remains a significant challenge. Existing Reinforcement Learning (RL) environments fail to address this specific market-level problem. Current tools like CityLearn focus on device-level scheduling (HVAC, batteries) for individual buildings, while theoretical utility pricing models often assume static compliance or risk-neutral objectives, failing to capture the stochastic behavioral fatigue of consumers facing repeated interventions. There is a lack of open-source, online environments that model the dynamic feedback loop between utility pricing signals and heterogeneous customer acceptance at the distribution level.

Methodology: DR-Gym Simulator Design
The authors introduce DR-Gym, an open-source, Gymnasium-compatible environment designed to train and evaluate demand-response policies from an electric utility's perspective. The system is modeled as an episodic Markov Decision Process (MDP) where the agent (utility) observes a 32-dimensional state vector and selects a continuous action (credit level in $/kWh) to maximize a configurable multi-objective reward.

Key architectural components include:

Building Demand Simulator: Utilizes physics-based hourly load profiles from the CityLearn/ResStock dataset (EnergyPlus simulations). A critical feature is demand-persistence feedback, where credits issued at time $t$ reduce loads in subsequent hours via exponential decay multipliers, preventing compounding reductions.
Wholesale Market Model: Decomposes prices into a Time-of-Use (TOU) base, an AR(1) persistence component with heteroscedastic noise, and a regime-switching spike model. This two-state Markov chain (Normal vs. Spike-storm) captures the temporal clustering of price spikes observed in real markets (e.g., ERCOT), calibrated to extreme events like Winter Storm Uri.
Heterogeneous Customer Response: Models customer acceptance as a logistic function of the credit level, extended to four distinct archetypes (Price-sensitive, Eco-conscious, Neutral, Reluctant) with varying base acceptance rates and sensitivities. Crucially, the model incorporates behavioral fatigue, where acceptance probabilities decay with consecutive activations and recover during non-activation steps.
Operational Budget & Reward: The utility operates under a daily budget constraint with rollover capabilities. The reward function is multi-objective, balancing utility revenue, consumer cost, grid stress, and an optional risk-aware penalty (e.g., Conditional Value-at-Risk, CVaR) to protect against tail-risk financial outcomes for consumers.

Key Contributions

First Standardized Utility-Level DR Environment: DR-Gym is presented as the first Gymnasium-compatible environment specifically for electric utility-level demand-response via credit issuance. It bridges the gap between device-level simulators and theoretical market models by providing a realistic, interactive testbed for single-agent RL.
Configurable Multi-Objective Reward System: The environment implements a flexible reward formulation that allows researchers to specify diverse learning objectives. This enables the training of agents to balance revenue optimization, grid stability, and consumer protection using "plug-and-play" risk-aware metrics.
Realistic Validation: The simulator is validated against real-world benchmarks, including CAISO/ERCOT day-ahead LMP statistics and empirical DR pilot data (e.g., Faruqui and Sergici), demonstrating realistic price volatility, demand patterns, and customer acceptance rates.

Experimental Results
The authors validate the learnability of DR-Gym using Proximal Policy Optimization (PPO) against four baseline heuristics (No Credit, Uniform Credit, Rule-Based, and Budget-Aware Rule).

Performance: The PPO agent achieved a statistically significant higher mean episode reward compared to all baselines.
Risk Mitigation: While the "No Credit" baseline offered no protection during spikes, and rule-based policies often exhausted budgets prematurely, the PPO agent learned to reserve credits for high-impact intervals. This resulted in an 18–24% reduction in the CVaR0.95 of consumer bills compared to the No Credit baseline across various volatility seeds.
Budget Efficiency: The learned policy utilized 72–85% of the daily budget, demonstrating selective rather than exhaustive credit issuance, while maintaining positive utility revenue.

Significance and Claims
The paper claims that DR-Gym fills a critical gap in the RL ecosystem by providing a general-purpose testbed for sequential decision-making in market-level utility settings. Unlike existing environments, it explicitly models consumer protection objectives under price uncertainty and behavioral fatigue. The authors modestly state that they do not claim state-of-the-art performance but demonstrate that the environment design produces a challenging yet tractable optimization problem where principled RL policies outperform rule-based heuristics. They position DR-Gym as a platform for future research in risk-neutral and risk-aware RL, multi-objective policy optimization, and equity-aware DR mechanism design. The authors emphasize that the environment is designed to study protective mechanisms, though they caution that reward weights must be carefully configured (prioritizing consumer welfare weights) to prevent agents from optimizing revenue at the expense of consumer protection.