🤖 machine learning

Short-Term Electricity Demand Forecasting for New England Using a Hybrid Transformer-XGBoost Framework with Weather, Calendar, and COVID-19 Indicators

This paper proposes a hybrid Transformer-XGBoost framework for short-term electricity demand forecasting in New England that, despite incorporating COVID-19 indicators and achieving high accuracy, demonstrates that such epidemiological features ultimately degrade test performance by amplifying overfitting to outdated pandemic patterns once behavioral adaptations occurred.

Original authors: Reza Ghanavati, Behrooz Mosallaei

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Reza Ghanavati, Behrooz Mosallaei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how much electricity the entire New England region will use tomorrow. It's a bit like trying to predict how much water a giant, thirsty garden will drink. You know the weather matters (hot days need more AC, cold days need more heat), and you know the day of the week matters (people use more power on weekdays than weekends). But what happens when the whole world suddenly changes, like during a pandemic? That's the puzzle this paper tries to solve.

Here is a simple breakdown of what the researchers did and what they found, using everyday analogies.

The Goal: Predicting the Power Grid's Thirst

Electricity is tricky because you can't really store it in a giant warehouse; you have to make exactly as much as people need, right now. If you guess wrong, the lights might flicker, or you might waste money making power nobody uses. The researchers wanted to build a "crystal ball" to predict daily electricity demand for New England with high accuracy.

The Recipe: A Hybrid Kitchen

The team cooked up a new method using a "hybrid" approach. Think of it like a restaurant with two chefs working together:

Chef 1 (The Transformer): This is a high-tech AI chef specialized in reading stories. It looks at the "story" of the last 30 days of electricity usage and pandemic news (like case counts and deaths). It's great at spotting long, complex patterns in a timeline, kind of like how a human can read a novel and understand the mood of a character over many chapters.
Chef 2 (XGBoost): This is a very sharp, logical chef who loves tables and spreadsheets. It looks at specific facts: "It's Tuesday," "The temperature in Boston is 40°F," and "We had 500 new cases yesterday." It's excellent at making quick decisions based on hard data.

The Hybrid Framework: Chef 1 reads the story and hands a summary note to Chef 2. Chef 2 then combines that summary with all the hard facts to make the final prediction. The researchers hoped this team would be better than just using Chef 2 alone.

The Ingredients

To make their prediction, they mixed in a lot of different data:

Weather: They didn't just look at one city; they checked the weather in six different cities across New England (like Boston, Portland, and Burlington) to get a full picture.
Calendar: They accounted for holidays, weekends, and seasons.
The Pandemic Factor: They included data about COVID-19 cases and deaths, hoping to capture how the virus changed people's habits (like working from home).
History: They looked at what happened yesterday, last week, and last month.

The Taste Test: Did It Work?

They tested their "Hybrid Kitchen" against a "Solo Kitchen" (just Chef 2/XGBoost without the AI story-reader).

The Result: The Hybrid Kitchen was slightly better at guessing the exact amount of electricity needed. However, when the researchers ran a strict statistical test (like a referee checking if the score difference was real or just luck), they found the two kitchens were essentially tied. The Hybrid model wasn't statistically better than the simple one.
The Takeaway: If you have a really good list of facts (weather, calendar, history), you don't necessarily need the fancy AI story-reader to get a great result. The simple, logical chef was almost just as good.

The Big Surprise: The Pandemic Ingredient

The most interesting part of the paper is what happened with the "Pandemic Ingredient" (COVID-19 data).

During the Pandemic (Training): When the model was learning from data from 2020–2021, the pandemic data was very useful. It helped the model understand why electricity use was weird (e.g., offices were empty, homes were full).
After the Pandemic (Testing): When they tested the model on data from late 2022 and early 2023, things got weird. By this time, people had adapted to the virus; we were back to work, and habits had normalized.
- The Glitch: The model thought the pandemic data was still super important. In fact, it relied on it more heavily during the test phase than it did during the learning phase!
- The Analogy: Imagine a student who learns that "rain means no soccer practice." They study hard during a rainy season. But then, the season changes, and it's sunny, yet the student still checks the rain forecast obsessively and cancels practice anyway. The model kept looking at the pandemic numbers as if the world was still in lockdown, even though people had already adapted.
- The Consequence: Because the model kept using this "stale" information, it actually made slightly worse predictions in the hybrid model when the pandemic data was included, compared to when it was removed.

The Final Verdict

History is King: The single most important thing for predicting electricity is knowing what happened yesterday and last week. If you remove that, the model fails miserably.
Weather is Queen: Knowing the temperature and the day of the week is the next most important thing.
The Pandemic is a "Time-Limited" Ingredient: The data about COVID-19 was helpful only while the world was in active lockdown. Once people adapted (by mid-2022), that data became "noise." It didn't help predict the future; it just confused the model by making it think the old rules still applied.

In short: The researchers built a smart system that works very well. They found that while fancy AI helps, a solid logical model with good weather and history data is almost just as good. Most importantly, they learned that when the world changes (like a pandemic), you have to be careful not to let the model get stuck in the past, using old rules that no longer apply.

Technical Summary: Short-Term Electricity Demand Forecasting for New England Using a Hybrid Transformer–XGBoost Framework

1. Problem Statement

Accurate short-term electricity demand forecasting is foundational for power system operators to manage real-time dispatch, unit commitment, and energy market clearing. While classical statistical methods (e.g., ARIMA) and modern machine learning approaches (e.g., XGBoost, LSTMs, Transformers) have been widely applied, forecasting remains challenging in regions with high weather sensitivity and structural demand disruptions.

This study addresses the specific challenges of the New England electricity market (ISO-NE), characterized by extreme seasonal weather variations and the structural behavioral shifts caused by the COVID-19 pandemic (2020–2022). A critical gap identified is the uncertainty regarding the predictive value of epidemiological indicators (case counts, death rates) as behavioral adaptation progresses from the acute pandemic phase to a post-acute stabilization period. The paper investigates whether a hybrid architecture combining deep learning for temporal feature extraction with gradient boosting for tabular prediction offers superior accuracy over a robust tabular-only baseline, and how the utility of pandemic-era features evolves over time.

2. Methodology

Data and Feature Engineering

The study utilizes daily electricity demand data for the New England control area from February 1, 2020, to March 23, 2023. The feature engineering pipeline integrates:

Meteorological Observations: 16 daily variables (max/avg/min temperature, dew point, humidity, wind speed, pressure, precipitation) from six cities representing all six New England states (Boston, Bridgeport, Burlington, Concord, Portland, Providence), yielding 96 raw weather features.
Calendar and Seasonal Indicators: Binary flags for holidays, weekends, workdays, and "bridge days," alongside continuous variables for day of week, day of year, and season.
Autoregressive Lags: Demand lags at 1, 7, 14, and 30 days to capture persistence and seasonality.
Epidemiological Indicators: Eight features derived from COVID-19 case and death counts, including raw daily counts, 7-day and 14-day rolling means, and daily percentage growth rates.

Model Architecture

The proposed framework employs a Hybrid Transformer–XGBoost architecture:

Transformer Encoder: Processes sliding windows (length $L=30$ ) of time-series channels (8 COVID features + 4 demand lags). It utilizes sinusoidal positional encoding, 4 attention heads, and two encoder layers to produce a 32-dimensional temporal embedding vector. This component is designed to capture long-range temporal dependencies and complex interaction patterns.
XGBoost Regressor: Takes the concatenation of the Transformer embedding vector and the full tabular feature set (weather, calendar, lags) as input. It is configured with shallow trees ( $max\_depth=3$ ) and strong regularization to prevent overfitting.

Training and Optimization

Data Split: A strict chronological split (70% training, 15% validation, 15% test) ensures no temporal leakage. The StandardScaler is fitted exclusively on the training set.
Hyperparameter Tuning: Bayesian optimization using Optuna with a multivariate Tree-structured Parzen Estimator (TPE) sampler over 500 trials. A MedianPruner discards unpromising trials early.
Evaluation Metrics: Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE), and the Coefficient of Determination ( $R^2$ ). Statistical significance of performance differences is assessed via the Diebold-Mariano (DM) test with Harvey-Leybourne-Newbold correction.
Interpretability: SHapley Additive exPlanations (SHAP) are used to quantify feature contributions and analyze temporal shifts in feature importance.

3. Key Results

Predictive Performance

Both models achieved high accuracy on the held-out test set (October 2022 – March 2023):

Hybrid Model: RMSE = 8,876 MWh, MAPE = 2.05%, $R^2$ = 0.906.
XGBoost-Only Baseline: RMSE = 9,304 MWh, MAPE = 2.21%, $R^2$ = 0.896.

While the hybrid model showed marginally better metrics, the Diebold-Mariano test ($DM = -1.126, p = 0.262$) confirmed that the 427.7 MWh difference in RMSE is statistically indistinguishable from noise. Thus, the two architectures are considered statistically equivalent in predictive accuracy for this dataset.

Feature Importance and Ablation

Dominance of Lags: Autoregressive demand lags (particularly 1-day lag) were the most critical features. Removing them increased test RMSE by ~15–18% in both models.
Weather and Calendar: These features alone explained approximately 84–85% of demand variance ( $R^2 \approx 0.84$ ).
Transformer Embeddings: Removing Transformer embeddings degraded the hybrid model's test RMSE by 7.4%, confirming they capture genuine temporal structure, though the marginal gain over the tabular baseline was modest.
COVID-19 Features (The Asymmetry):
- Training Period: COVID features improved training accuracy (removing them increased training RMSE by 2.5% for XGBoost-only).
- Test Period: The effect was asymmetric. Removing COVID features improved the XGBoost-only test RMSE by 1.2% but degraded the hybrid test RMSE by 3.2%.
- SHAP Analysis: A critical finding revealed that 5 of the 8 COVID features ranked higher (were deemed more important) by the hybrid model on the post-acute test set than during the pandemic-active training set. This indicates the model over-applied learned pandemic patterns to a period where behavioral responses had normalized, leading to systematic misfires.

4. Significance and Claims

The paper establishes that while hybrid deep learning architectures can capture temporal dynamics, they do not necessarily yield statistically significant accuracy gains over well-tuned gradient boosting models when rich tabular features are available.

The central contribution of this work is the identification of temporal validity decay as a mechanism for forecasting error. The study demonstrates that epidemiological features, while highly informative during the acute disruption phase (2020–2021), became counterproductive noise in the post-acute phase (late 2022–2023) because the behavioral drivers (lockdowns, remote work mandates) that linked viral metrics to electricity demand had dissipated.

The authors conclude that:

Operational Parsimony: For daily demand forecasting in this context, a tabular-only XGBoost model is a competitive operational choice, offering similar accuracy to the hybrid model with lower implementation complexity.
Feature Lifecycle Management: Epidemiological features should be time-gated or retired once behavioral adaptation is complete. Relying on them after the pandemic's acute phase leads to overfitting on stale patterns, a phenomenon detectable via SHAP temporal analysis where feature importance increases despite predictive utility decreasing.
Behavioral vs. Virological Drivers: The study clarifies that the demand response to COVID was driven by behavioral disruptions rather than viral transmission rates alone; once behavior normalized, the correlation between case counts and demand broke down, regardless of ongoing viral circulation.