Predicting Heterogeneous Treatment Effects Of Building Energy Saving Retrofits Using Causal Machine Learning

Technical Summary: Predicting Heterogeneous Treatment Effects of Building Energy Saving Retrofits Using Causal Machine Learning

Problem Statement
Information Systems (IS) research increasingly utilizes machine learning (ML) for predictive modeling in complex socio-technical systems. However, standard predictive ML optimizes for correlational accuracy rather than causal identification, rendering it unsuitable for estimating the impact of interventions where treatment assignment is non-random. This limitation is critical in building energy retrofits, where the decision to renovate is heavily biased by household characteristics (e.g., income, location) and building attributes (e.g., age, size) that simultaneously influence energy consumption. Consequently, standard models often yield biased estimates of energy savings, failing to distinguish between the effect of the retrofit and the underlying confounding factors. While physical simulations exist, they are resource-intensive and often diverge from realized savings due to unmodeled occupant behaviors. Data-driven approaches offer scalability but typically lack causal rigor. The core challenge is to accurately recover Heterogeneous Treatment Effects (HTEs)—the varying impact of retrofits on individual buildings—under conditions of strong selection bias.

Methodology
To address the impossibility of observing ground-truth counterfactuals in real-world observational data, the authors constructed a physically grounded simulation environment. This approach allows for the explicit modeling of true treatment effects and empirically motivated adoption biases, creating a dataset with known ground truth for rigorous benchmarking.

• Data Generation: The simulation adapts thermodynamic formulas (Kirschbaum et al., 2025) to model heat loss and annual heating requirements. It generates a synthetic dataset of single-family homes across various construction periods (1850–1994). The dataset includes four distinct retrofit interventions: windows only; windows and roof; exterior walls only; and a comprehensive package (windows, roof, and exterior walls).
• Bias Mechanism: Treatment assignment is deliberately biased to reflect real-world adoption patterns. The probability of receiving a retrofit is modeled as a function of socio-economic determinants (income), building stock characteristics (age, size), and climatic conditions. This creates genuine confounding, where variables influencing the treatment also directly affect the outcome (energy consumption).
• Assumptions: The study assumes the Stable Unit Treatment Value Assumption (SUTVA) holds by design, as buildings do not interfere with one another. The unconfoundedness assumption is satisfied by construction, as all relevant confounders are included in the simulation. The overlap (common support) assumption is verified through propensity score distributions, showing adequate overlap across treatment groups.
• Models Evaluated: The study benchmarks several causal ML estimators against a naive Ordinary Least Squares (OLS) baseline:
  • Metalearners: S-Learner, T-Learner, and X-Learner, which utilize flexible ML algorithms (Random Forests) to estimate Conditional Average Treatment Effects (CATEs).
  • Double Machine Learning (DoubleML): A debiased approach that orthogonalizes the treatment and outcome models to remove confounding bias, utilizing cross-fitting and residual regression.
• Evaluation: Models are trained on the synthetic data and evaluated based on Mean Absolute Error (MAE) and normalized MAE against the known true CATEs.

Key Results
The comparative analysis reveals distinct performance hierarchies among the estimators, particularly as retrofit complexity increases:

• Overall Performance: OLS exhibited the highest error (nMAE: 62.0%), confirming that ignoring confounding leads to severe bias. Among causal estimators, DoubleML achieved the lowest average error (nMAE: 11.6%), followed by the T-Learner (11.4%) and X-Learner (11.5%). The S-Learner performed the worst among the causal methods (13.7%).
• Treatment Complexity: For the simplest intervention (windows only), the X-Learner was the most accurate. However, as the complexity and magnitude of the true treatment effects increased (e.g., exterior wall retrofits and comprehensive packages), DoubleML significantly outperformed all metalearners. For the comprehensive retrofit, DoubleML's error was 170.13 lower than the T-Learner and 479.82 lower than the X-Learner.
• Extrapolation and Generalization: Scatterplot analysis indicated that metalearners (S, T, X) tended to attenuate large treatment effects, failing to predict extreme energy savings beyond the range of the training data. This is attributed to the tree-based base learners' inability to extrapolate beyond observed leaf-node averages. In contrast, DoubleML tracked the ground truth closely across all effect sizes, including the high-saving tail, due to its final-stage linear regression on residuals which is not bound by training data ranges.
• Sample Size Sensitivity: Performance improved with sample size for all models. DoubleML required a larger sample (approx. 10,000 total observations) to outperform metalearners but consistently achieved the lowest error thereafter.
• Confounding Control: SHAP analysis demonstrated that metalearners retained residual influence from confounding variables (e.g., construction year) in their CATE estimates, whereas DoubleML effectively removed this influence through orthogonalization.

Significance and Contributions
The paper makes three primary contributions to the IS and energy informatics literature:

1. Quantitative Assessment: It provides the first quantitative assessment of causal ML estimator accuracy for individual house retrofit scenarios with known bias. Unlike previous empirical studies limited to Average Treatment Effects (ATE) or plausibility checks, this study validates performance against known HTEs.
2. Systematic Benchmarking: It offers a systematic comparison of OLS, metalearners, and DoubleML, identifying that while metalearners are competitive for simple interventions, DoubleML is superior for complex, high-confounding scenarios typical of deep energy retrofits.
3. Methodological Foundation: The study demonstrates that orthogonalizing treatment assignment (as in DoubleML) significantly improves causal effect estimation in biased observational settings. This provides a methodological basis for using causal ML in large-scale energy retrofit policy evaluation, offering a scalable alternative to physical simulations while maintaining accuracy.

Limitations and Future Work
The authors acknowledge that the simulation, while grounded in physical laws and empirical bias patterns, does not capture the full spectrum of real-world complexities, such as behavioral rebound effects, capital constraints, or detailed occupant interactions. The study also relies exclusively on Random Forests as base learners and does not include other causal models like R-Learners or Causal Forests. Future research is suggested to incorporate these behavioral dynamics, test alternative base learners, and validate findings against real-world observational datasets to strengthen external validity.