⚡ electrical engineering

Safe Deep Reinforcement Learning for Building Heating Control and Demand-side Flexibility

This paper proposes a safe deep reinforcement learning framework utilizing a deep deterministic policy gradient algorithm and a real-time adaptive safety filter to optimize building heating control, achieving significant energy and cost savings while ensuring strict compliance with demand-side flexibility requests and maintaining occupant comfort.

Original authors: Colin Jüni, Mina Montazeri, Yi Guo, Federica Bellizio, Giovanni Sansavini, Philipp Heer

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Colin Jüni, Mina Montazeri, Yi Guo, Federica Bellizio, Giovanni Sansavini, Philipp Heer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your home's heating system is like a chef cooking a complex meal for a family. The chef's job is to keep the food (the room temperature) perfectly warm, but they also have to deal with two tricky things:

The Price of Ingredients: Sometimes electricity is cheap (like buying veggies on sale), and sometimes it's expensive (like buying truffles).
The Special Order: The power grid (the "restaurant manager") might call and say, "Hey, we need you to cook less for the next two hours because we have too much solar power right now," or "Cook a bit more because we need power."

The Problem: The Old Chef vs. The New Chef

The Old Way (Rule-Based Control):
Think of the old heating system as a chef who only follows a simple rule: "If the soup is cold, turn on the stove. If it's hot, turn it off."

Pros: It's simple.
Cons: It doesn't care if electricity is expensive right now. It doesn't listen to the "restaurant manager." It just keeps the soup warm, wasting money and missing chances to help the grid.

The New Way (Deep Reinforcement Learning - DRL):
Now, imagine a super-smart AI chef. This chef learns by trial and error. It tastes the soup, checks the price of ingredients, and learns that "If I heat the soup a little extra when the price is low, I can turn the stove off later when the price is high."

Pros: It saves a lot of money and keeps the soup at the perfect temperature.
Cons: Because it's learning by experimenting, it might get too greedy. It might try to save money by turning the stove off too early, making the soup cold (uncomfortable for the family). Or, it might ignore the "restaurant manager's" special order, which could cause problems for the whole power grid.

The Solution: The "Safety Filter"

This is where the paper's big idea comes in. The authors built a Safety Filter, which acts like a strict but helpful sous-chef standing right next to the AI chef.

Here is how the sous-chef works:

The AI Chef makes a plan: "I'm going to turn the stove down to save money!"
The Sous-Chef checks the rules: "Wait! The 'restaurant manager' just asked us to reduce cooking by 30% for the next hour. If you turn it down that much, we won't meet the order."
The Adjustment: The sous-chef says, "Okay, you can turn it down, but only this much. Not more."
The Result: The AI chef still gets to be smart and save money, but the final action is guaranteed to be safe and follow the rules.

Why is this special?

Most safety systems in the past were like rigid rulebooks that required a perfect map of the building's physics (like knowing exactly how much heat the walls hold). If the building was old or weird, the map was wrong, and the system failed.

This new Safety Filter is adaptive. It doesn't need a perfect map. It looks at the current situation in real-time:

"Is the room getting too cold?" -> Loosen the rules a bit so the AI can heat up.
"Is electricity super cheap?" -> Let the AI use a bit more energy to save money later.
"Is the 'manager' asking for a big cut?" -> Tighten the rules immediately.

The Results: A Happy Chef, a Happy Family, and a Happy Grid

The authors tested this in a real apartment building (a "living lab" in Switzerland). Here is what happened:

Compared to the Old Chef: The new system saved about 50% on energy bills. That's like cutting your grocery bill in half!
Compared to the AI Chef without a Sous-Chef: The AI alone was great at saving money but sometimes broke the "manager's" rules. With the Safety Filter, it followed the rules 100% of the time without losing much of its money-saving magic.
Comfort: The family stayed warm. The only downside was a tiny increase in moments where the room was slightly cooler than perfect, but it was a fair trade for the massive savings and grid stability.

The Big Picture

Think of the power grid as a busy highway.

Renewable energy (wind/solar) is like a sudden rush of cars.
Buildings are like cars that can speed up or slow down.
The Safety Filter is the traffic cop that tells the AI driver (the building) exactly when to slow down or speed up so no one crashes, while still letting the driver take the most efficient route.

This paper proves that we can have smart, self-learning buildings that save us money and help the planet, as long as we give them a safety net to make sure they don't go off the rails.

1. Problem Statement

The building sector accounts for approximately 40% of global energy consumption and a significant portion of CO2 emissions. As the energy grid integrates more intermittent renewable energy sources (RES), there is an urgent need for Demand-Side Flexibility (DSF), where buildings adjust their energy consumption to support grid stability.

However, controlling building Heating, Ventilation, and Air Conditioning (HVAC) systems to provide this flexibility presents three main challenges:

Safety and Constraints: Standard Reinforcement Learning (RL) agents are exploratory and may violate critical constraints, such as occupant thermal comfort or specific energy reduction targets requested by grid operators.
Model Dependency: Traditional Model Predictive Control (MPC) requires accurate physical models of the building, which are difficult to obtain and scale across diverse building types.
Real-time Adaptability: Existing solutions often rely on static rules or complex system identification, limiting their ability to adapt dynamically to fluctuating electricity prices and real-time flexibility requests.

The core problem is to design a control framework that optimizes heating for cost and comfort while guaranteeing strict compliance with external flexibility constraints (e.g., reducing energy use by a specific factor during a specific time window) without relying on pre-defined physical models.

2. Methodology

The authors propose a Safe Deep Reinforcement Learning (SRL) framework consisting of three main components:

A. Deep Reinforcement Learning Controller (DDPG)

Algorithm: The core controller uses the Deep Deterministic Policy Gradient (DDPG) algorithm, suitable for continuous action spaces (valve opening levels).
State Space ( $S_t$ ): Includes solar irradiation, ambient temperature, room temperature, neighboring room temperature, time features, electricity price, flexibility window timing, and cumulative energy usage.
Action: Continuous valve opening level ( $u_t$ ) for the heat pump.
Reward Function: A weighted sum designed to balance three objectives:
- Thermal comfort (penalizing deviations from $23.5^\circ\text{C} - 25^\circ\text{C}$ ).
- Electricity cost (minimizing $u_t \times \text{price}$ ).
- Flexibility compliance (penalizing deviations from the requested energy budget).
Thermal Model: The environment uses a Physically Consistent Neural Network (PCNN). This hybrid model combines a linear physical model with a non-linear neural network to capture unforced dynamics, ensuring accurate temperature prediction without needing a full first-principles model.

B. Real-time Adaptive Safety Filter (RASF)

To address the safety limitations of standard DRL, a model-free, real-time safety filter is introduced. This filter sits between the DRL agent and the building environment.

Function: It takes the proposed action ( $u_t$ ) from the DRL agent and adjusts it to a safe action ( $u_{safe,t}$ ) to ensure the cumulative energy consumption within a flexibility window does not exceed the operator's request.
Mechanism:
1. Baseline Calculation: Computes the remaining average safe action ( $u_{1,t}$ ) required to meet the energy budget over the remaining time.
2. Dynamic Tolerance ( $\tau_t$ ): Calculates a tolerance factor based on two normalized indicators:
  - Thermal Flexibility: How close the room temperature is to comfort limits (higher tolerance if the room is far from limits).
  - Price Favorability: How low the electricity price is (higher tolerance if prices are low).
3. Action Adjustment: The filter defines upper and lower bounds for the valve opening based on $u_{1,t}$ $u_{1, t}$ and $\tau_t$ $τ_{t}$ .
  - If energy must be reduced: $u_{safe,t} = \min(u_t, u_{maximal,t})$ .
  - If energy must be increased: $u_{safe,t} = \max(u_t, u_{minimal,t})$ .

C. Simulation Environment

The system was tested using historical data from the UMAR apartment unit at the NEST building (Empa, Switzerland). The simulation covers a one-month period with 15-minute resolution, comparing the proposed method against Rule-Based Control (RBC) and standard DRL.

3. Key Contributions

Model-Free Adaptive Safety Filter: A novel RASF that requires no prior system identification or physical modeling. It dynamically adjusts control actions in real-time based on room temperature and electricity prices to guarantee compliance with flexibility requests.
Integrated Safe DRL Framework: The successful integration of the safety filter into a DDPG-based control architecture, enabling the agent to learn optimal policies while strictly adhering to external grid constraints.
Real-World Validation: A close-to-reality simulation using data from a living laboratory building, demonstrating the controller's ability to optimize energy usage, maintain thermal comfort, and deliver flexibility services simultaneously.

4. Results

The study compared four scenarios:

Baseline: Rule-Based Control (hysteresis).
DRL (No Flex): Optimizes cost/comfort but ignores flexibility requests.
DRL (Flex): Optimizes for flexibility but lacks the safety filter (risk of violation).
DRL + RASF: The proposed method.

Key Findings (One-Month Simulation):

Cost Efficiency: The DRL + RASF approach achieved a 64.5% cost reduction compared to the Rule-Based baseline (CHF 19.42 vs. CHF 40.75).
Energy Savings: It reduced energy consumption by nearly 50% compared to the baseline (69.02 kWh vs. 134.62 kWh).
Flexibility Compliance: Unlike the standard DRL (Case 3), which failed to meet some flexibility requests, the RASF ensured 100% compliance with operator constraints.
Comfort Trade-off: While enforcing flexibility slightly increased comfort violations (from 75.62 Kh in "No Flex" to 103.56 Kh in "Flex + RASF"), the system still significantly outperformed the baseline (97.12 Kh) in overall thermal stability and cost.
Performance vs. Baseline: The proposed controller reduced costs by 64.5% and improved thermal comfort metrics by 45.5% compared to the Rule-Based controller.

5. Significance

This paper addresses a critical gap in smart grid research: the lack of guaranteed safety in data-driven building control.

Grid Stability: By ensuring buildings strictly adhere to flexibility requests, the method allows buildings to act as reliable, active participants in grid balancing and congestion management.
Scalability: The model-free nature of the approach (relying on PCNNs and data rather than complex physics equations) makes it highly scalable to diverse building types where accurate physical models are unavailable.
Economic Viability: The results demonstrate that strict safety constraints do not preclude economic benefits; the system achieves massive cost savings while satisfying grid operators.
Future Impact: The framework provides a blueprint for deploying AI in critical infrastructure where safety violations (e.g., discomfort or grid instability) are unacceptable, paving the way for decentralized, resilient energy systems.