Understanding electricity consumption behaviour through Inverse Reinforcement Learning

Technical Summary: Understanding Electricity Consumption Behaviour through Inverse Reinforcement Learning

Problem Statement
Residential electricity consumption is a critical component of global energy demand, particularly regarding space cooling, which is increasing due to climate change and geopolitical energy crises. A significant challenge in energy policy design is understanding how households respond to socioeconomic and climatic shocks (e.g., price spikes and heatwaves). Existing approaches often oversimplify consumer behavior, relying on stylized "white box" models that lack granularity or machine learning methods that focus on limited subpopulations, thereby hindering generalizability. Furthermore, current literature lacks a quantitative understanding of how cooling behaviors evolve under combined thermal and socioeconomic stress, particularly at sub-daily temporal resolutions. The core problem is how to model household consumption at a large scale for broad consumer groups while avoiding oversimplification, maintaining interpretability, and capturing the non-linear, heterogeneous responses to shocks.

Methodology
The authors propose a framework based on Adversarial Inverse Reinforcement Learning (AIRL) to infer reward functions that represent household consumption behavior. The methodology proceeds through three main stages:

1. Data Preprocessing and Clustering:

   • Data Source: Hourly smart meter data from Italian households (May–July 2021–2023), covering the pre-crisis, crisis, and post-crisis periods.
   • Context Variables: Static/quasi-static variables (income, urbanization, building vintage, estimated AC ownership, Green View Index, billed energy price) and dynamic variables (UTCI thermal stress index, consumption profiles).
   • Clustering: To handle heterogeneity, the authors employ a clustering pipeline. Static variables are standardized, while dynamic time-series data (consumption and UTCI) are processed using Echo State Networks (ESNs) and Tensor PCA to extract low-dimensional embeddings that preserve temporal dynamics. These are concatenated (50% weight each) and clustered using agglomerative hierarchical clustering (Ward linkage).
   • Selection: 11 clusters were identified. For detailed analysis, four representative clusters were selected: high-consumption, urban intermediate-consumption, urban low-consumption, and rural low-consumption.

2. Adversarial Inverse Reinforcement Learning (AIRL):

   • Formulation: Households are treated as agents in a Markov Decision Process (MDP). The state $s$ includes context variables (UTCI, time of day, day type, rolling means) and the action $a$ is defined as the first-order difference in consumption ($e_{t+1} - e_t$).
   • Algorithm: An AIRL algorithm (specifically using Proximal Policy Optimization for the generator) is trained for each cluster. The system consists of a generator policy and a discriminator (reward network).
   • Reward Disentanglement: The reward function $r(s, a)$ is decomposed into a base term $f_\phi(s, a)$ and a potential-based shaping term $\Phi_\psi(s)$. This structure ensures the extracted reward is disentangled from environment dynamics, allowing for transferability across changing conditions (e.g., from 2021 to 2023).
   • Training: The model is trained to maximize the likelihood of the generator being classified as an expert (the real data). Hyperparameters are optimized using Bayesian optimization to minimize a Combined Metric (CM) that balances distribution similarity, consumption error, and temporal correlation.

3. Behavior Analysis:

   • Once trained, the reward networks are interrogated to extract behavioral insights.
   • Optimal Consumption: A softmax distribution over consumption values is constructed based on the learned reward to visualize the "optimal" consumption states agents would choose under different thermal stress levels.
   • Comparison: These model-implied optimal curves are compared against empirical average consumption to identify shifts in behavior (persistent, transient, or negligible).
   • Sub-clustering: To investigate intraday heterogeneity, the high-consumption cluster was further sub-clustered based on consumption timing (afternoon-high vs. evening-high) to isolate the effect of daily activity patterns.

Key Results
The study analyzed Italian household data across the summers of 2021, 2022, and 2023, encompassing the energy crisis and a severe heatwave in 2022.

• Heterogeneous Responses to Shocks: The reward functions revealed that shocks reshaped cooling behavior differently across groups:

  • High-Consumption Cluster: Exhibited a persistent shift. While they reacted strongly to temperature in 2021, the 2022 crisis led to a durable reduction in consumption that did not fully recover in 2023.
  • Urban Intermediate-Consumption Cluster: Showed a transient shift. They amplified their response to heat in 2022 but largely reverted to 2021 trends in 2023 as conditions eased.
  • Rural Low-Consumption Cluster: Displayed a structural, slow shift. Initially flat, they developed a more temperature-responsive behavior in 2022 that consolidated in 2023, suggesting a latent potential for cooling adoption.
  • Urban Low-Consumption Cluster: Exhibited negligible change, remaining the least responsive group, likely constrained by the built environment and budget.

• Intraday Heterogeneity: Sub-clustering the high-consumption group by timing revealed distinct behaviors:

  • Afternoon-High: Users with higher AC ownership and daytime activity reduced consumption significantly during the 2022 crisis peak.
  • Evening-High: Users with evening peaks were more reluctant to change in the short term, maintaining high-UTCI responses in 2022 before converging to the afternoon group's behavior in 2023.
  • This indicates that time-of-use is a separate dimension of behavioral heterogeneity, independent of socioeconomic status.

Significance and Claims
The paper claims three primary contributions to the literature:

1. Methodological Innovation: It introduces an AIRL framework to infer reward-implied behavioral response surfaces from large-scale smart meter data. Unlike traditional empirical correlations, this approach represents households as agents whose consumption is shaped by a disentangled reward function, capturing non-linear interactions between thermal, socioeconomic, and temporal contexts.
2. Empirical Insight into Crisis Dynamics: By applying the framework to the 2021–2023 Italian energy crisis, the study quantifies how cooling behaviors change under concurrent thermal and socioeconomic stress. It distinguishes between transient adjustments and durable shifts, providing a granular view of how different consumer groups adapt (or fail to adapt) to extreme conditions.
3. Revealing Hidden Heterogeneity: The study demonstrates that behavioral differences exist even within groups that appear similar in socioeconomic and environmental terms. Specifically, it identifies the timing of consumption (intraday patterns) as a critical factor that influences how households respond to shocks, a dimension often overlooked in aggregate analyses.

Limitations and Future Work
The authors acknowledge limitations regarding the granularity of context variables (e.g., income and urbanization are aggregated at the municipal/zipcode level) and the partial identifiability of the reward function (absolute scales are not intrinsic). They suggest future work could incorporate higher-resolution socioeconomic data, utilize Long Short-Term Memory (LSTM) architectures for better intraday trend reproduction, and explore higher-order partial derivatives of the reward function. The study also notes that specific contract changes and energy poverty profiles were not explicitly modeled, suggesting these as areas for future integration.