🤖 machine learning

A Unified Python Framework for Direct PPO-based Control of AHUs with Economizer Logic and CO2-Constrained Ventilation

This paper presents a unified Python framework utilizing Proximal Policy Optimization (PPO) to control Air Handling Units with integrated economizer logic and a hierarchical flow mechanism that overrides agent actions to maintain CO2 levels below 1000 ppm, demonstrating superior temperature stability and energy efficiency compared to traditional PID and On-Off controllers.

Original authors: Erfan Haghighat Damavandi, Davide Papurello, Mahdi Alibeigi, Armin Keshavarz, Simone Canevarolo, Marco Condo

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Erfan Haghighat Damavandi, Davide Papurello, Mahdi Alibeigi, Armin Keshavarz, Simone Canevarolo, Marco Condo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large office building as a giant, living house that needs to breathe, stay cool, and keep its occupants comfortable. The paper you're reading is about teaching this "house" how to manage its own air conditioning system (called an AHU) using a smart, self-learning computer brain instead of a simple, dumb thermostat.

Here is the breakdown of their approach, using simple analogies:

The Problem: The "Dumb" Thermostat

Currently, most buildings use two types of controllers:

The On/Off Switch: Like a light switch that is either fully ON or fully OFF. If the room gets too hot, the AC blasts at full power until it's cold, then shuts off completely. This causes the room temperature to swing wildly (like a pendulum), making people uncomfortable and wasting energy.
The PID Controller: A slightly smarter version that tries to smooth out those swings, but it still follows rigid rules and doesn't adapt well to changes, like a sudden crowd of people entering the room.

The problem is that buildings are complex. The temperature changes, the humidity shifts, and the number of people (who breathe out CO2) changes constantly. Traditional controllers struggle to balance comfort (not too hot/cold), air quality (not too much CO2), and energy savings all at once.

The Solution: A "Learning Student" (PPO)

The authors created a new system using Deep Reinforcement Learning (DRL), specifically an algorithm called PPO.

Think of the PPO agent not as a rule-following robot, but as a student learning to drive.

The Environment: The student is placed in a simulation of a building (a "digital twin") that includes physics like heat transfer and how CO2 builds up when people talk.
The Goal: The student's only job is to keep the room at a perfect 22°C (71.6°F).
The Reward: Every time the room stays close to 22°C, the student gets a "good job" point. Every time the temperature drifts away, they get a "bad job" penalty.
The Learning: Through trial and error over 300,000 simulated minutes, the student learns exactly how much to turn the fan on or off to keep the temperature perfect, even when the weather outside changes or a crowd of people enters.

The Special Features: The "Safety Net" and the "Free Lunch"

The paper introduces two clever tricks to make this student even smarter:

1. The Hierarchical Flow Logic (The Safety Net)
In a normal learning scenario, the student might try to turn the fan off completely to save energy, which would cause the CO2 levels to spike and make people sick.

The Fix: The authors built a "Safety Net" into the system. Even if the student tries to turn the fan off, the Safety Net overrides that decision if the CO2 levels get too high (above 1000 ppm). It calculates exactly how much fresh air is needed to keep the air safe and forces the fan to run at least that much. This ensures the building never gets "stuffy," no matter how much the student tries to save energy.

2. The Economizer (The Free Lunch)
Sometimes, the air outside is cooler and drier than the air inside.

The Fix: The system checks the outside air. If it's "free cooling" (like a cool breeze on a summer day), the system opens the windows (or dampers) wide to let that free air in, rather than using electricity to run the heavy cooling coils. It's like opening a window on a cool evening instead of turning on the AC.

The Results: Who Won the Race?

The authors tested four different "drivers" in the same building over a 24-hour period:

The On/Off Thermostat: The room temperature bounced up and down like a yo-yo. CO2 levels sometimes got dangerously high.
The PID Controller: Much smoother temperature control, but still used a bit too much energy.
The Basic PPO Student: Did a great job keeping the temperature steady, similar to the PID.
The Smart PPO Student (with the Safety Net and Free Lunch): This was the winner.
- Comfort: It kept the temperature rock-steady at 22°C.
- Air Quality: It kept CO2 levels perfectly below the safety limit (around 910 ppm) without wasting air.
- Energy: It used about 6% less energy than the other controllers. It did this by knowing exactly when to bring in free outside air and when to minimize airflow when the building was empty.

The Bottom Line

The paper claims that by using a unified Python framework with this specific "Smart Student" (PPO) and its built-in safety rules, a building can be cooler, have fresher air, and use less electricity than with traditional methods. They have built a complete simulation tool that proves this works, and they plan to test it on a real building next.

Technical Summary: A Unified Python Framework for Direct PPO-based Control of AHUs

Problem Statement

Building HVAC systems are responsible for a significant portion of global energy consumption (30–40% of building energy use), yet maintaining optimal performance is challenging due to the nonlinear characteristics of building envelopes and stochastic load variations. Traditional control strategies, such as On/Off thermostats and fixed-setpoint PID controllers, often struggle to balance thermal comfort, Indoor Air Quality (IAQ), and energy efficiency simultaneously. Specifically, these conventional methods frequently fail to dynamically adjust ventilation rates based on real-time occupancy and CO2 levels, leading to either excessive energy waste from over-ventilation or health risks from poor IAQ. Furthermore, existing Deep Reinforcement Learning (DRL) studies often treat IAQ as a soft penalty rather than a hard constraint, rely on complex co-simulation tools that hinder implementation, or lack integrated physical models (such as thermal networks and mass balance equations) within a unified Python environment.

Methodology

The authors propose a unified Python framework that implements a Direct Proximal Policy Optimization (PPO) algorithm to control an Air Handling Unit (AHU). The methodology is built upon the following components:

1. Physical and Mathematical Modeling

The simulation environment utilizes a 2R-2C thermal network model coupled with dynamic mass balance equations for moisture and CO2.

Thermal Dynamics: Modeled using thermal resistance ( $R_{in}, R_{out}$ ) and capacitance ( $C_{wall}$ ) to simulate transient thermodynamic behavior.
Moisture and CO2: The system calculates indoor humidity and CO2 concentration based on occupant generation rates, supply air properties, and ventilation rates.
Economizer Logic: An enthalpy-based economizer is integrated to enable "free cooling" when outdoor conditions (temperature and humidity) are more favorable than indoor conditions.

2. Control Strategies Compared

The study evaluates four distinct control strategies over a 24-hour dynamic simulation:

On/Off Thermostat: A baseline switching logic with a fixed differential (hysteresis).
PID Controller: A continuous control signal modulating fan speed based on proportional, integral, and derivative gains.
Standard PPO Agent: An RL agent trained to regulate temperature with a fixed fresh air ratio (50%).
Proposed "Smart" PPO Agent: An advanced agent featuring Hierarchical Flow Logic.

3. The Proposed PPO Framework

The core innovation lies in the Hierarchical Flow Logic and the environment design:

State Space: The agent observes the current indoor temperature ( $T_{air}$ ) and the instantaneous thermal error ( $e(t)$ ).
Action Space: The agent outputs a continuous action to control the total airflow rate.
Hierarchical Flow Logic (IAQ Override): This is a critical safety mechanism. The environment calculates the minimum fresh air volume required to keep CO2 below 1000 ppm based on real-time occupancy. The final actuation is the maximum of the agent's requested flow and the calculated IAQ minimum flow. This ensures the agent cannot shut off ventilation to save energy if it would compromise air quality.
Reward Function: The agent is rewarded based on the negative squared error of the temperature deviation from the 22°C setpoint ( $R = -|T_{air} - T_{set}|^2$ ).
Training: The agent is trained using the Stable Baselines3 library over 300,000 timesteps with dynamic hourly disturbances.

Key Contributions

The paper identifies three primary contributions to the field of HVAC control:

Self-Contained Gym Environment: A custom Python environment that couples a 2R-2C thermal model with a dynamic CO2 balance, eliminating the need for external co-simulation tools.
Hard IAQ Constraints: Unlike many RL studies that treat IAQ as a soft penalty, this framework enforces IAQ as a hard constraint via the Hierarchical Flow Logic, ensuring CO2 levels never exceed the 1000 ppm safety limit.
Integrated Optimization: The system successfully balances the sole objective of temperature regulation with the physical necessity of ventilation, demonstrating that an RL agent can learn to optimize energy use without sacrificing occupant health.

Results

The simulation results, comparing the four strategies over a 24-hour period with dynamic occupancy and weather profiles, indicate:

Thermal Comfort: Both the PID and PPO controllers significantly outperformed the On/Off thermostat, eliminating the large temperature oscillations (sawtooth pattern) associated with binary switching. The PPO agents achieved stability comparable to the PID controller.
Indoor Air Quality (CO2):
- The On/Off thermostat resulted in frequent CO2 spikes exceeding 1300 ppm.
- The PID controller maintained CO2 around 920 ppm but showed occasional limit exceedances during occupancy drops.
- The Smart PPO with Economizer Logic demonstrated the highest precision, maintaining CO2 levels strictly below 910 ppm during peak demand and tracking the 1000 ppm threshold accurately to prevent over-ventilation.
Energy Efficiency:
- On/Off: 2776 kWh
- PID: 2760 kWh
- Standard PPO: 2762 kWh
- Smart PPO (with Economizer/DCV): 2609 kWh
  The Smart PPO strategy achieved approximately 6% energy savings compared to the PID and On/Off baselines while simultaneously achieving the lowest CO2 levels.

Significance and Claims

The paper claims that this research addresses significant gaps in current HVAC control literature by providing a robust, generalized, and easily implementable Python-based solution. The significance of the work is twofold:

Practical Implementation: By integrating physical models (2R-2C, mass balance) directly into a Gym-style environment, the framework offers a pathway for "end-to-end" solutions that can be more easily transferred to real hardware implementations compared to complex co-simulation approaches.
Safety and Efficiency: The Hierarchical Flow Logic proves that RL agents can be constrained to prioritize safety (IAQ) without compromising their ability to learn optimal energy policies. The study demonstrates that dynamic, demand-controlled ventilation driven by RL can reduce energy consumption while strictly adhering to health standards, offering a viable alternative to traditional rule-based or reactive control systems.

The authors conclude that while the current work is a simulation, the framework is designed to be deployed on real physical AHUs, with future work planned to validate these findings using actual building data and real-world hardware deployment.