Technical Summary: Building2Building (B2B)

Problem Statement

Reinforcement Learning (RL) has demonstrated strong performance in control tasks, yet learned policies often remain brittle when faced with changes in dynamics, action spaces, observation spaces, or goals. This lack of generalization is a critical barrier to real-world deployment. Existing benchmarks for studying generalization, such as Meta-World and RLBench, are largely confined to robotics and video games. They often introduce limited diversity, focusing primarily on dynamic parameter variations (e.g., friction) or task goals, while rarely offering heterogeneous observation and action spaces or complex physical environments. Consequently, research into cross-domain adaptation and transfer in RL often relies on custom, non-reproducible experimental setups, fragmenting progress.

The authors identify Heating, Ventilation, and Air Conditioning (HVAC) control in buildings as an ideal domain to address these gaps. Buildings exhibit vast diversity in layouts, materials, climate exposure, and HVAC configurations, yet they share common physical principles based on thermodynamics and energy conservation. Furthermore, HVAC systems represent a significant societal challenge, accounting for nearly half of the energy consumption in buildings (which comprise roughly one-third of global energy use).

Methodology

The Building2Building (B2B) Suite

The authors introduce Building2Building (B2B), a large-scale suite of realistic HVAC control environments built on EnergyPlus, a state-of-the-art building energy simulator. B2B is designed to be fully compatible with the Gymnasium interface.

Environment Generation:

• Scale: The suite generates a dataset of 6,000 environments spanning multiple building types (houses, restaurants, warehouses, retail stores, small/medium offices), climates, and HVAC configurations.
• Generators:
  1. A residential generator producing single-zone houses based on the statistical characteristics of the Québec residential building stock.
  2. A parametric generator based on ASHRAE 90.1 reference commercial buildings across 16 North American locations. This generator samples building size, window-to-wall ratios, insulation, and air infiltration while enforcing climate-dependent parameter bounds to ensure physical consistency.
• Control Interface: B2B exposes low-level HVAC actuators, handling:
  • Unitary systems: Control of Supply Air Temperature (SAT) and supply air flow rate.
  • Central air loop systems: Control of central SAT, outdoor-air mixers, zone damper openings, and zone-level reheat coils.
  • Baseboard/Heating-only systems: Control via zone thermostats.
• Observations: Agents receive weather data (temperature, humidity), calendar information, indoor zone temperatures, and normalized HVAC energy consumption.

Structured Environment Representation

To address the lack of standard ways to describe heterogeneous fleets of environments, B2B introduces a structured representation that extends the standard Gymnasium API.

• Morphological Universe: Defines a common vocabulary for a family of environments, including a common attribute space ($C_c$), observation space ($S_c$), and action space ($A_c$), alongside a set of node types ($T$) with local spaces.
• Morphology: Describes a specific environment instance as a graph where nodes represent components (e.g., thermal zones, HVAC equipment) with specific local attributes.
• Decomposition: Observations and actions are decomposed into a common part and a morphological graph part. This allows algorithms (e.g., Graph Neural Networks, Heterogeneous Graph Transformers) to operate on the structure of the environment rather than fixed input/output sizes.

Benchmark Tasks

The authors define four reproducible benchmark settings to isolate specific sources of variation:

1. Goal Adaptation: Training and testing on the same building dynamics but with varying reward functions (e.g., changing energy weights $w_E$ or temperature setpoint strategies: constant, occupancy-based, or random).
2. Dynamics Adaptation: Fixed reward and action space, but varying building dynamics (climate, size, envelope) within the same archetype.
3. Action-Space Shifts: Fixed dynamics and reward, but changing the set of controllable actuators between training and testing (e.g., training on airflow only, testing on airflow + SAT).
4. Cross-Domain Generalization: Training and testing on entirely different building types with heterogeneous observation and action spaces (e.g., training on retail stores, testing on small offices).

Reward Formulation

The reward function balances thermal comfort and energy efficiency:
$$r_t = -\frac{1}{Z \tau_T} \sum_{z=1}^{Z} (T_z(t) - T^{target}_z(t))^2 - w_E \frac{1}{\tau_E} E_{HVAC}(t)$$
To handle scale differences across thousands of environments, the authors introduce normalizing constants $\tau_T$ and $\tau_E$. $\tau_E$ is fixed as the mean per-step power consumption of a baseline reactive policy for a given building type and climate, allowing a single weight $w_E$ to define the trade-off across diverse environments.

Key Results

Baselines:
The authors established baselines using:

1. Reactive Control: Supervisory control logic inspired by ASHRAE Guideline 36 (PI loops and Trim-and-Respond mechanisms).
2. PPO Agents: Trained on subsets of the test set.

Dynamics Adaptation Experiments:

• Three model classes were tested on single-zone houses: (1) Per-building specialists, (2) Non-specialist agents (resampling buildings), and (3) Parameterized agents (resampling + building attributes).
• Findings: Per-building specialists achieved the best performance, but there was a substantial gap compared to non-specialists. Parameterized agents outperformed non-specialists, demonstrating the benefit of conditioning on building parameters (floor area, construction year, etc.) for dynamics adaptation. However, non-specialist and parameterized models showed "long tail" failures on specific test buildings, whereas specialists were consistent.

Cross-Domain Transfer Experiments:

• A single policy was trained simultaneously on four building types (retail, restaurant, small office, medium office) using a variant of the Amorpheus architecture (type-specific encoders/decoders with a shared transformer).
• Findings: The generalized policy successfully adapted to unseen buildings from the test set. In several cases, it outperformed the reactive baseline controller, demonstrating the feasibility of learning control policies across heterogeneous morphologies without per-building retraining.

Significance and Claims

The paper claims that B2B provides a necessary large-scale, diverse, and physically grounded testbed for the systematic investigation of generalization and transfer in continuous control. Its significance lies in three areas:

1. Advancing RL Research: By offering a standardized framework with heterogeneous observation and action spaces, B2B enables rigorous study of multi-task learning, meta-learning, and cross-domain transfer, moving beyond the limitations of current robotics-centric benchmarks.
2. Real-World Applicability: The benchmark bridges the gap between fundamental RL research and real-world deployment in energy management. It supports a new training paradigm where policies are trained across a distribution of buildings rather than requiring a highly detailed digital twin for every individual building.
3. Societal Impact: By enabling improved HVAC control at scale, the work contributes directly to reducing energy consumption in buildings, lowering operational costs, and aiding climate change mitigation.

The authors emphasize that while current RL methods show promise in generalization, the B2B benchmark reveals that significant challenges remain, particularly in handling the "long tail" of failure cases when transferring to unseen environments without specific conditioning.