⚡ electrical engineering

Comparative Field Deployment of Reinforcement Learning and Model Predictive Control for Residential HVAC

This paper presents a one-month field deployment comparing Model Predictive Control (MPC) and Model-Based Reinforcement Learning (RL) for residential HVAC, demonstrating that while both achieved similar energy savings (18.1% and 20.9% respectively) with comparable data needs, RL required significantly less engineering effort but initially compromised occupant comfort due to safety and initialization challenges.

Original authors: Ozan Baris Mulayim, Elias N. Pergantis, Levi D. Reyes Premer, Bingqing Chen, Guannan Qu, Kevin J. Kircher, Mario Bergés

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ozan Baris Mulayim, Elias N. Pergantis, Levi D. Reyes Premer, Bingqing Chen, Guannan Qu, Kevin J. Kircher, Mario Bergés

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your home's heating and cooling system as a very eager, but slightly dim-witted, robot butler. Its only job is to keep the house at a temperature you set on a dial. If you tell it to stay at 20°C, it will blast the heater until it hits 20°C, then shut off, only to turn it back on when the house cools down. This "set it and forget it" method works, but it's wasteful. It's like driving a car with your foot glued to the gas pedal, only tapping the brakes when you're about to crash, rather than slowing down smoothly before the stop sign.

To fix this, scientists have been trying to teach the robot to be smarter. There are two main ways to do this. The first is like giving the robot a detailed map and a physics textbook. This is called Model Predictive Control (MPC). The robot uses the map to calculate exactly how the house will heat up or cool down over the next day, planning its moves perfectly to save energy while keeping you cozy. The second way is Reinforcement Learning (RL). Instead of a map, you give the robot a reward system: "Good job if you save energy, bad job if you make the house too cold." The robot learns by trial and error, figuring out the best moves by watching what happens, much like a video game character learning to beat a level by dying repeatedly until it gets the pattern right.

The big question is: Which method is better for a real house? The textbook method (MPC) is proven but requires a human engineer to build the map and tune the rules, which takes a lot of time and effort. The learning method (RL) promises to do the work automatically, but no one was sure if it could learn fast enough without making the house freezing cold or wasting energy while it "figured things out." This paper takes these two approaches out of the computer simulation lab and drops them into a real, occupied house in a cold climate to see which one actually works in the wild.

The Great Thermostat Showdown: Maps vs. Learning

In a cozy, 1920s-era house in West Lafayette, Indiana, two digital brains went head-to-head for a month. One brain was the Model Predictive Control (MPC) system, the "planner" that relies on a pre-built model of how the house reacts to heat. The other was Ibex-RL, a Reinforcement Learning system, the "learner" that tries to figure out the house's behavior on the fly while it's running. Both were tasked with controlling a heat pump—the heart of the home's heating system—to save electricity while keeping the residents comfortable.

The Scoreboard: Who Saved More Energy?
The results were surprisingly close, like two runners finishing a race within a hair's breadth of each other.

The MPC planner managed to save 18.1% of the energy compared to the standard, constant-temperature setting.
The RL learner actually edged it out slightly, saving 20.9% of the energy.

Both methods were statistically proven to be better than doing nothing, but the difference between the two robots was not significant enough to declare a clear winner in terms of pure energy savings.

The Comfort Factor: The "Too Cold" Problem
Here is where the two robots had very different personalities.
The MPC planner was the cautious, rule-following type. It stuck very closely to the temperature the residents wanted (18°C at night, 20°C during the day). The residents were happy, and the system maintained "acceptable" comfort levels the whole time.

The RL learner, however, was a bit of a risk-taker. It learned that keeping the house slightly cooler saved more energy. As a result, it kept the house cooler than the residents preferred, especially during the first few days while it was still "learning." This led to three reports of discomfort from the people living there. The house got chilly enough that the residents actually complained. While the RL system eventually adjusted, it spent more time in the "too cold" zone than the MPC system did.

The Effort: How Hard Was It to Set Up?
This is where the story gets interesting for anyone trying to build these systems.

MPC is like building a custom suit. It requires a tailor (an engineer) to take measurements, fit the fabric, and adjust the seams. The researchers estimated that setting up MPC for a new house would take about five days of an engineer's time just for the control logic, plus four days for the physical setup.
RL is like buying a smart, self-adjusting jacket. Once the basic framework is there, it adapts to the wearer. The researchers estimated that setting up RL for a new house would take only about two days of control work.

In short, the RL system required about one-third less engineering effort to deploy than the MPC system. It was faster to get up and running, but it came with the initial "learning curve" of making the residents a bit chilly.

The Glitches and The "Black Box"
The experiment wasn't without its hiccups. The RL system had a bit of a identity crisis. It was supposed to learn the physics of the house (how fast the walls cool down, how much sun comes through the windows), but because it was learning from real-world data without a perfect "simulator" to practice on first, some of its learned numbers got weird. For instance, it briefly thought that sunlight cooled the house down (a negative solar area), which is physically impossible. However, the system was built with safety rails (constraints) that prevented these weird ideas from causing the heater to go haywire. It kept the house safe, even if its internal map was a little distorted.

The Verdict
This study suggests that Reinforcement Learning is a viable path for making homes smarter and saving energy, potentially saving engineers a lot of time in the process. However, it also highlights that RL isn't a magic wand yet. It can be a bit "rough around the edges" at the start, risking occupant comfort while it figures things out.

The paper concludes that while RL offers a faster, more automated way to deploy smart controls, it still faces challenges in ensuring safety and comfort during its learning phase. The ideal future might not be a choice between the two, but a hybrid: using the safety and structure of the "planner" (MPC) to guide the "learner" (RL), creating a system that is both easy to set up and gentle on the residents' comfort. For now, if you want the absolute safest bet for a new home, the planner (MPC) is still the reliable choice. If you want to try the future of automation and are willing to tolerate a few chilly nights while it learns, the learner (RL) is ready to give it a shot.

Technical Summary: Comparative Field Deployment of Model-Based Reinforcement Learning and Model Predictive Control for Residential HVAC

Problem Statement

Model Predictive Control (MPC) has demonstrated significant potential for improving energy efficiency in residential Heating, Ventilation, and Air Conditioning (HVAC) systems compared to standard control methods. However, deploying MPC often requires substantial engineering effort, including the manual tuning of physics-based models and cost functions. Reinforcement Learning (RL) offers a promising alternative that could automate strategy discovery and reduce modeling burdens. Yet, the practical application of RL in residential settings remains largely undemonstrated. Critical gaps exist regarding occupant comfort during the learning phase, data requirements, and the safety of online adaptation. Furthermore, most RL research relies on high-fidelity simulators, creating a bottleneck that negates the goal of scalable, "simulator-free" deployment. There is a lack of empirical, long-term comparative studies evaluating the trade-offs between MPC and RL in real, occupied homes.

Methodology

To address these gaps, the authors conducted a month-long field deployment in a single occupied detached house in West Lafayette, Indiana, during a cold winter period. The study compared two supervisory control algorithms for an air-to-air heat pump with staged electric resistance backup:

MPC Variant: A data-driven MPC approach that learns a physics-informed system dynamics model (2R1C thermal circuit) via offline system identification. It uses a Support Vector Machine (SVM) to predict exogenous heat gains. The controller optimizes a cost function balancing energy costs and thermal discomfort, with online tuning of the discomfort weight to maintain occupant satisfaction.
RL Variant (Ibex-RL): A model-based RL approach based on the Differentiable MPC policy framework. Unlike standard model-free RL, Ibex-RL learns a physics-informed system dynamics model and a quadratic cost function simultaneously. It employs a hybrid offline-to-online strategy:
- Offline Phase: Uses imitation learning on historical data to initialize system dynamics and cost parameters, ensuring a "safe but suboptimal" starting point without requiring a high-fidelity simulator.
- Online Phase: Continuously adapts both the dynamics model ( $\theta_{state}$ ) and cost parameters ( $\theta_{cost}$ ) using gradient-based updates derived from a non-quadratic reward signal that mirrors the MPC objective.

Experimental Setup:

Control Action: Both controllers adjusted the thermostat temperature setpoint (sent via API) based on indoor temperature and electric power measurements.
Constraints: Both utilized a 2R1C thermal circuit model. A hard safety constraint prevented setpoints below 17°C.
Baseline: Performance was benchmarked against a constant-setpoint baseline (occupant-preferred schedule) and a non-supervised operation period.
Data: Training and validation utilized historical data from different time periods (2022 for MPC, 2023 for RL). Evaluation data was collected during the 30-day deployment (Jan–Feb 2025).

Key Contributions

The paper makes three primary contributions to the literature:

First Empirical Comparison: It provides the first direct, side-by-side comparison of MPC and model-based RL in an occupied residential house, moving beyond simulation studies.
Long-Term RL Deployment: It documents a month-long field test of a model-based RL algorithm (Ibex-RL) without the safety net of a pre-trained high-fidelity simulator, offering insights into long-term adaptation and stability.
Practical Roadmap: It outlines specific lessons learned regarding model accuracy, safety assurance, and engineering effort, providing a realistic assessment of the challenges in deploying advanced controllers "in the wild."

Results

Energy Savings

Relative to a constant-setpoint baseline, both controllers achieved significant weather-normalized energy savings:

MPC: Saved 18.1% (95% CI: 4.4% to 30.9%).
RL (Ibex-RL): Saved 20.9% (95% CI: 2.6% to 38.3%).
Statistical analysis indicated no significant difference in energy savings between the two algorithms. When normalized by the indoor-outdoor temperature difference (isolating equipment efficiency from setpoint reduction), MPC showed slightly higher efficiency (11.28% savings) compared to RL (8.04%), though the difference was not statistically significant.

Occupant Comfort

MPC: Maintained acceptable comfort levels, with a time-average Predicted Percentage of Dissatisfied (PPD) of 9.92%, staying within the typical 10% acceptability threshold. No discomfort reports were received.
RL: Received three reports of occupant discomfort. The house was generally cooler than the user preference, particularly during an initial adaptation phase and on specific cold days. The time-average PPD was higher at 13.53%. The RL controller kept the house cooler to maximize energy savings, prioritizing the reward signal over strict setpoint tracking during the learning phase.

Engineering Effort and Scalability

The authors estimated the recurring engineering effort required to deploy these specific algorithms in a new house (assuming existing infrastructure and historical data):

MPC: Approximately 5 engineer-days (for model fitting, data preprocessing, and tuning).
RL: Approximately 2 engineer-days (for imitation learning setup and hyperparameter tuning).
The authors estimate that a fresh RL deployment would require about one-third less engineering effort than MPC. However, they note that the initial development of the RL algorithm required significant modification of the Gnu-RL framework (approx. 45 student-days) and faced unique maintenance challenges, such as API communication failures that corrupted online learning data.

Significance and Claims

The paper claims that while RL reduces deployment effort and achieves comparable energy savings to MPC, it faces distinct challenges related to safe controller initialization and mismatches between modeled and true state/action spaces.

The authors emphasize that their findings are specific to the implemented variants (Ibex-RL and the specific MPC implementation) and should not be viewed as a general verdict on all RL or MPC methods. They argue that the study highlights a critical trade-off: RL offers a pathway to scalable, automated control with lower recurring engineering costs, but this comes with risks to occupant comfort during the adaptation phase and requires robust safety layers to handle real-world data integrity issues.

Ultimately, the paper suggests that the future of advanced HVAC control lies not in a competition between paradigms, but in their convergence. The authors posit that hybrid solutions—fusing the automated learning and adaptation of RL within the interpretable, constraint-aware framework of MPC—may be the most viable path toward scalable, real-world deployability.