Learning to Modulate, Not to Cycle: Soft Actor---Critic Recovers Inverter-Style Heat-Pump Control

Technical Summary: Learning to Modulate, Not to Cycle

Problem Statement
Residential heat pumps, particularly single-stage units, suffer from significant mechanical wear and reduced efficiency due to frequent on–off cycling. Every compressor start incurs transient losses (refrigerant migration, oil return, pressure equalisation) that degrade the Coefficient of Performance (COP) and accelerates component failure. While Inverter-driven (variable-capacity) heat pumps naturally suppress cycling by modulating speed to track loads, standard Reinforcement Learning (RL) controllers for building HVAC typically optimize only a weighted sum of thermal comfort and energy cost. These standard RL approaches often ignore the cycling behavior of the learned policy, potentially resulting in controllers that achieve good cost–comfort scores but "chatter" the compressor in ways that real hardware cannot sustain.

Methodology
The authors formulate single-zone heat-pump control as a discrete-time Markov Decision Process (MDP) with a 15-minute time step, utilizing the "bestest hydronic heat pump" test case from the BOPTEST framework.

• Action Space: The agent controls a normalized compressor capacity $u_{HP} \in [0, 1]$. This continuous space allows the agent to theoretically behave as either a single-stage thermostat (switching between 0 and 1) or an inverter (modulating continuously between endpoints).
• Reward Design: A novel reward function is introduced that augments standard comfort and cost terms with a levelised compressor-wear term. This term monetizes each compressor start-up based on a representative replacement cost amortized over the rated number of starts. Crucially, this term is expressed in the same currency (euros) as the energy cost, allowing the agent to trade off wear against operating costs directly.
• Training Environment: Agents are trained on a fast, data-driven surrogate plant identified via Sparse Identification of Nonlinear Dynamics (SINDy) to reduce computational cost. Policies are subsequently evaluated on the high-fidelity BOPTEST emulator.
• Algorithms Compared: The study compares two model-free algorithms under identical reward structures and observation spaces:
  • Soft Actor–Critic (SAC): An off-policy, maximum-entropy actor–critic method.
  • Proximal Policy Optimisation (PPO): An on-policy clipped policy-gradient method.

Key Contributions

1. Smoothing-Aware Reward: The paper introduces a physically grounded, monetised compressor-wear term that makes switching a "first-class" objective without requiring explicit minimum-runtime constraints.
2. Algorithm-Class Effect: The study demonstrates that the choice of RL algorithm fundamentally dictates the qualitative form of the control signal. Even with identical environments and rewards, SAC learns a continuous modulation policy, while PPO collapses to a bang–bang policy.
3. Validated Trade-off: The work provides empirical evidence that a smooth SAC policy can significantly reduce thermal discomfort and eliminate cycling on the BOPTEST emulator, with only a modest increase in operating cost.

Results

• Cycling Behavior: On both peak and typical heating scenarios, the SAC agent achieved zero compressor starts per day, maintaining the compressor in a permanently engaged, modulated state (inverter-style operation). In contrast, the PPO agent learned a bang–bang policy that cycled 2 to 4 times more often than the baseline controller, effectively worsening the wear problem despite the wear penalty in the reward.
• Performance Metrics:
  • Comfort vs. Cost: On the peak heating scenario, the SAC policy reduced thermal discomfort by 90.7% (from 8.38 to 0.78 K·h) for an 11.5% increase in operating cost.
  • Energy Usage: The cost increase for SAC was driven by higher source energy consumption (11.0% more than baseline), indicating the agent delivered more heat to maintain comfort rather than exploiting tariff timing.
  • PPO Performance: PPO reduced costs by under-heating the zone but failed to improve comfort significantly and induced excessive cycling.
• Control Signal: SAC's control signal varied continuously (never dropping to zero), whereas PPO spent 60–82% of steps fully off, saturating at the boundaries.

Significance and Claims
The paper argues that for equipment where lifetime is governed by switching, entropy-regularised off-policy learning (SAC) is structurally aligned with the smoothness hardware requires. The maximum-entropy objective of SAC values a broad action distribution and penalizes collapse onto deterministic rails (0 or 1), naturally guiding the agent toward the "always-on" modulation optimum. Conversely, PPO, lacking this entropy floor, finds it easier to raise average duty by mixing fully-on and fully-off actions.

The authors conclude that by pricing each start-up in the reward function, equivalent smoothness to engineered inverter controllers can arise endogenously from the learning process. The study suggests that SAC offers a desirable Pareto point: near-zero discomfort and zero wear for a bounded, recoverable cost premium. The work does not claim to replace hardware inverters but demonstrates that RL can learn to operate single-stage hardware with the beneficial modulation characteristics of inverters, provided the algorithm and reward structure are correctly aligned. Future work is limited to sweeping the wear weight to trace the Pareto frontier and benchmarking against Model Predictive Control (MPC).