Data Leakage Inflates Generalizability of Power Outage Prediction Models

When a storm rolls in, the lights go out. For utility companies and emergency responders, knowing exactly where and when those lights will fail is a matter of public safety. Power outages disrupt water supplies, communication networks, and hospitals, turning a bad weather day into a crisis that can last for days or even weeks. To prevent this, scientists have spent the last fifteen years building computer models designed to predict these blackouts. These models act like weather forecasters for the electrical grid, using data on wind, rain, tree cover, and population density to guess which neighborhoods will lose power. The hope has been that these models can learn from past storms to prepare for future ones, especially as climate change makes extreme weather more frequent and severe.

However, a new study suggests that the confidence we have in these predictions might be misplaced. Researchers from the University of Toronto and North Carolina State University investigated whether these models can truly handle the unknown. They found that many of the impressive success stories reported in scientific literature are likely illusions created by a flaw in how the models are tested. The core problem is that the tests often let the model "use information it should not have," making the predictions look much better than they really are. When the researchers forced the models to face truly new situations—like a storm in a state they had never seen before, or a storm type they had never encountered—their ability to predict outages collapsed, often failing to do better than a simple guess.

The researchers focused on the United States East Coast, a region battered by hurricanes, winter storms, and thunderstorms. They gathered data on power outages from 2018 to 2023, matching every reported blackout with weather conditions and land features. To test the models, they tried three different ways of splitting the data into training and testing groups. The first method, which is the standard in most previous studies, is a random shuffle. Imagine taking a deck of cards, mixing them thoroughly, and dealing half to the model to learn from and half to test it. Because weather patterns are connected across space and time, a random shuffle often puts very similar days or nearby counties into both the learning and testing groups. The model essentially memorizes the neighborhood rather than learning the rules of the storm.

The second and third methods were much stricter. In the first strict test, the researchers removed an entire state from the training data and asked the model to predict outages only for that state. This simulates a scenario where a model trained in one region is deployed in a completely new one. In the second strict test, they removed an entire storm event from the training data, asking the model to predict the impact of a specific hurricane or winter storm it had never seen before. These tests mimic the real world, where a utility company might face a new type of disaster in a new location with no prior history of that specific event.

When the models were tested using the random shuffle method, they performed well, explaining about 45 percent of the variation in outage percentages. This result aligns with what other studies have claimed. But when the researchers applied the stricter tests, the performance dropped dramatically. In the state-by-state tests, the models often performed no better than a "null model," which is a baseline that simply guesses the average number of outages for that area every time. For predicting the absolute number of customers without power, the models failed to outperform this simple guess in almost every case. Even for predicting the percentage of customers affected, the models struggled significantly, with their accuracy varying wildly depending on the state or the storm.

The study also examined whether using advanced artificial intelligence could fix the problem. The researchers tried feeding the models "embeddings" from a powerful foundation model called Prithvi WxC. This model is trained on global weather data and is designed to understand complex patterns in the atmosphere. The hope was that this high-tech approach would help the models generalize better. While the foundation model did offer a slight improvement in some spatial tests, it did not solve the fundamental issue. When faced with a new storm event, the advanced model performed just as poorly as the simpler ones, and in some cases, it performed worse. The researchers found that the high complexity of the foundation model's data actually made it harder for the model to learn the specific patterns needed for new events, leading to overfitting where the model memorized the training data rather than understanding the underlying physics.

The findings suggest that the current generation of power outage prediction models, when trained on publicly available data, offers limited value for real-world disaster planning. The models are not failing because they lack sophisticated algorithms; they are failing because the data they are trained on does not contain enough diverse examples of extreme events to teach them how to handle the unknown. The relationship between a storm and a power outage is not a simple rule that applies everywhere. A hurricane that knocks out power in Florida behaves differently than a winter storm in New Jersey, and a model trained on one cannot easily predict the other. The study indicates that without better data coverage, more realistic testing methods, and a shift away from trying to squeeze marginal improvements out of existing datasets, these models will remain unreliable for the critical task of protecting communities from the next big storm.