📊 statistics

Intrinsic predictability rather than urban structure governs machine learning forecast skill for electricity demand

This paper argues that the forecast skill of machine learning models for urban electricity demand is primarily determined by the intrinsic predictability of a city's historical load patterns (specifically seasonal variance and temperature sensitivity) rather than by the city's structural characteristics or geographic location.

Original authors: Mahyar Hassani, Ataollah Rafiei Atani, Mahdi Abdolhamid, Mohammadreza Hamidizadeh

Published 2026-09-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mahyar Hassani, Ataollah Rafiei Atani, Mahdi Abdolhamid, Mohammadreza Hamidizadeh

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Cities are increasingly turning to artificial intelligence to manage their electricity needs, hoping that smart algorithms can predict exactly how much power people will use next month. This promise of precision is central to modern urban planning, where officials must balance the grid, cut costs, and reduce emissions. The logic seems straightforward: if a computer can forecast demand accurately, it proves the city's energy system is well understood; if the forecast fails, it suggests something hidden or broken is happening within that specific city's infrastructure or behavior. This assumption has driven a wave of investment in machine learning tools for public administration, treating prediction errors as direct clues to local problems that need fixing. However, this approach rests on a critical, often unexamined belief: that the ability of a model to predict the future depends on the unique structure of the city itself.

A new study challenges this fundamental belief by looking at electricity demand across forty-six large cities in the United States. The researchers, led by Mahyar Hassani and colleagues, did not simply build a better forecasting engine; they built a diagnostic tool to test whether the engine's success or failure actually tells us anything about the city. They analyzed twelve years of monthly electricity data, from 2007 to 2018, applying a rigorous testing method that prevents the computer from using future data. Their goal was to see if the accuracy of the predictions varied because of how big a city was, how wealthy it was, or how its buildings were arranged, or if the variation came from something else entirely.

The results were surprising. The study found that the skill of the machine learning model had almost nothing to do with the city's size, its population, or its specific urban layout. A city with a million people was not inherently easier or harder to predict than a city with a quarter of a million. Instead, the ability to forecast demand was governed almost entirely by the city's own history of electricity use, specifically two simple patterns: how strongly the demand followed the seasons and how tightly it tracked the temperature. Cities with a strong, predictable rhythm of heating in winter and cooling in summer were easy to forecast, regardless of whether they were in the desert or the Midwest. Cities with milder climates, where the weather and electricity use did not swing dramatically from month to month, were much harder to predict. The researchers found that these two factors—the seasonal rhythm and the temperature sensitivity—explained nearly all the differences in forecast accuracy between the cities.

To reach this conclusion, the team had to strip away several layers of potential confusion. They compared their advanced machine learning models against a very simple rule: guessing that next month's electricity use would be the same as the same month last year. In nearly half of the cities, this simple "seasonal-naïve" rule performed just as well as the complex artificial intelligence. In some cases, the simple rule was actually better. This showed that for many cities, the "signal" of the seasons was so strong that no amount of computer power could add much value. For the cities where the complex models did better, the improvement was often marginal. The study demonstrated that when a model fails to predict a city's demand, it is rarely because the city is chaotic or poorly governed. Instead, the failure is usually because the city's electricity use simply lacks a strong, predictable pattern to begin with.

The researchers also uncovered a hidden danger in how data is handled. They showed that a common, routine practice of filling in missing data points with a repeated value could create a fake geographic pattern. If a few cities happened to have missing data that was filled in this way, the models would suddenly appear to fail in a specific region, creating the illusion of a regional problem where none existed. This finding serves as a stark warning: what looks like a deep insight into urban governance might just be an artifact of how the numbers were cleaned. By controlling for these data quirks and the intrinsic predictability of the demand, the study found no evidence that forecast skill clusters by region or that it reveals hidden local failures.

The practical takeaway for city officials is a shift in how they interpret these forecasts. A poor prediction is not, on its own, evidence of a broken system or a hidden local crisis worth investigating. It becomes evidence of a problem only after the city's inherent predictability has been accounted for. If a city has a flat, unseasonal demand pattern, a low forecast score is expected and tells the official nothing new. If a city has a strong seasonal pattern but the model still fails, that is when the alarm should ring. The study suggests that before spending money on audits or new programs to fix "bad" forecasts, administrators should first check if the city's demand history actually allows for a good forecast. By focusing on the intrinsic nature of the data rather than the complexity of the algorithm, public officials can avoid chasing ghosts and focus their resources on the cities and systems where real problems actually exist.

Technical Summary: Intrinsic Predictability Rather Than Urban Structure Governs Machine Learning Forecast Skill for Electricity Demand

Problem Statement
Public administrators increasingly rely on machine learning (ML) forecasts of urban energy demand to guide policy and infrastructure planning. However, a critical gap exists in understanding when a forecast's performance serves as valid evidence about the city it describes versus when it merely reflects the algorithm's limitations or the data's inherent properties. Conventional approaches often treat predictive failure as a direct diagnostic of unmodeled local urban dynamics (e.g., governance inefficiencies or structural anomalies). This paper argues that such interpretations are frequently erroneous because they fail to account for "intrinsic predictability"—the theoretical ceiling of forecast skill imposed by the signal-to-noise ratio and temporal structure of the demand series itself. Without controlling for this ceiling, low forecast accuracy cannot be distinguished from irreducible process noise, leading to potential misallocation of audit resources and incorrect policy diagnoses.

Methodology: The Policy Learning Framework
The authors propose a "Policy Learning Framework" that reinterprets ML models not merely as predictive instruments but as diagnostic tools for systemic discovery. The methodology employs a two-stage diagnostic protocol applied to monthly residential electricity demand across 46 large United States cities (2007–2018).

Data and Features: The study utilizes a balanced urban panel (Obringer et al.) containing 12 years of monthly data. The feature set includes 26 ex-ante variables: trigonometric encodings of seasonality, socio-demographic covariates (population), contemporaneous and lagged meteorological variables (temperature, precipitation, humidity, etc.), and autoregressive lags of electricity and water consumption.
Leakage-Audited Walk-Forward Protocol: To ensure temporal validity and prevent data leakage, the authors implement an expanding-window walk-forward cross-validation. Models are trained on an initial 72 months and forecast one month ahead, recursively updating the training set. Crucially, the protocol includes algebraic guards to prevent target leakage (e.g., ensuring rolling means do not mathematically reconstruct the target) and strict separation of training and testing scalers.
Predictive Engine: The primary engine is XGBoost, evaluated against a "model ladder" of seven classes: seasonal-naïve, Ordinary Least Squares (OLS), Random Forest, Gradient Boosting, CatBoost (with and without ordered boosting), and a global DeepAR-style LSTM.
Diagnostic Scorecard: Instead of raw accuracy, the framework generates a diagnostic scorecard for each city. This scorecard compares realized out-of-sample skill ( $R^2$ $R^{2}$ ) against:
- A seasonal-naïve baseline.
- An ex-ante estimate of intrinsic predictability, derived from the seasonal share of load variance and temperature sensitivity calculated strictly on pre-forecast historical data.
Cross-Sectional and Spatial Analysis: The study regresses forecast skill against intrinsic predictability metrics and a suite of urban covariates (size, morphology, socio-economic factors). Spatial autocorrelation is tested using Global Moran's I across nine distinct weight matrices to determine if forecast skill exhibits geographic clustering independent of climate.

Key Results

Dominance of Intrinsic Predictability: Forecast skill varies widely across cities (from $R^2 \approx 0.27$ to $0.97$). However, this variation is almost entirely explained by two ex-ante properties of the demand history: the seasonal share of load variance and the sensitivity of load to temperature. Together, these account for an adjusted $R^2$ of 0.454 in explaining cross-city skill differences.
Irrelevance of Urban Structure: Contrary to the hypothesis that urban morphology or governance drives forecastability, city size (population) shows no correlation with skill ( $r = +0.017, p = 0.91$ ). Adding high-dimensional socio-economic covariates (Tier B) yields no marginal predictive value and exhibits signs of overfitting.
Spatial Null Hypothesis: Once intrinsic predictability and thermal climate are controlled for, forecast skill shows no residual spatial autocorrelation across any of the nine weight definitions. Geographic patterns in raw skill are entirely explained by the spatial smoothness of climate, not by regional governance or structural similarities.
Model Performance: The complex XGBoost engine ( $R^2 = 0.760$ ) outperforms the seasonal-naïve rule ( $R^2 = 0.740$ ) in only 28 of 46 cities. CatBoost achieved the highest mean $R^2$ (0.791), but the marginal gains over simpler ensembles were small and not statistically significant in all cities.
Data-Handling Artifacts: A factorial experiment revealed that routine data-handling choices can manufacture spurious geographic patterns. Specifically, "constant-fill" imputation of missing months (a common shortcut) artificially created a significant spatial cluster of forecast failures in Midwestern cities, whereas the correct approach (listwise deletion) did not. Similarly, target leakage via trailing-inclusive rolling means inflated accuracy globally but did not create spatial artifacts.

Significance and Claims
The paper claims to establish a "stopping rule" for urban energy governance: predictive failure is informative about governance only once intrinsic predictability has been controlled for.

Epistemological Shift: The study reframes forecasting from a confirmation exercise to a Popperian falsification process. A low $R^2$ is not evidence of a broken city or poor governance; it is often a property of the signal's inherent noise. Only a significant shortfall relative to the intrinsic predictability ceiling constitutes "Informative Failure" warranting investigation into unmodeled local dynamics.
Policy Implications: Public administrators should not commission audits or interventions based solely on poor forecast performance. Instead, they must first assess a city's intrinsic predictability. For cities with low seasonal signals (e.g., coastal cities with mild climates), low forecast skill is expected and does not imply latent structural problems.
Methodological Contribution: The paper highlights the "reproducibility crisis" in urban analytics, demonstrating how standard data-handling practices (imputation, duplicated utility territories, leakage) can generate convincing but entirely false geographic patterns. The proposed "Diagnostic Scorecard" serves as a boundary object, translating complex model outputs into transparent, non-maladaptive guidance for policymakers by explicitly reporting uncertainty, benchmarks, and predictability ceilings.

In conclusion, the authors assert that the variation in machine learning forecast skill for electricity demand is governed by the intrinsic statistical properties of the demand series rather than the urban structure of the city. This finding necessitates a shift in how predictive models are evaluated and utilized in public administration, prioritizing diagnostic rigor over raw accuracy.

More like this