Quantifying advances in infectious disease forecast skill across seasons and pathogens
This study introduces a novel skill score metric comparing forecasts against hindcast and baseline benchmarks to demonstrate that ensemble and team forecasts for influenza and COVID-19 have generally improved in skill over the past decade, despite mixed results for COVID-19.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer
Imagine trying to predict the weather a month from now. You know the general patterns of the seasons, but the specific path of a storm, the exact day it will rain, or how hard the wind will blow remains a guess. Infectious disease forecasting works much the same way. Instead of clouds and fronts, scientists track viruses and bacteria, using data on how many people are getting sick, how many are in the hospital, and how many are dying. The goal is to give public health officials a head start, allowing them to stock up on medicines, prepare hospital beds, and warn communities before a wave of illness hits. For over a decade, researchers in the United States have been running a massive, collaborative experiment to see if they can get better at these predictions. They have gathered thousands of forecasts from dozens of different teams, covering everything from the seasonal flu to the global pandemic of COVID-19. But a critical question has lingered: are these forecasts actually getting better over time, or are they just spinning their wheels?
To answer this, a team of researchers from Northeastern University and the Centers for Disease Control and Prevention developed a new way to measure success. In the past, scientists compared forecasts using scores that told them which model was better than another for a specific week, but these scores were hard to compare across different diseases or different years. A score that looked good for a mild flu season might look terrible for a chaotic pandemic year, not because the model failed, but because the situation itself was harder to predict. The researchers needed a ruler that could measure progress fairly, regardless of how messy the data was. They created a "skill score," a metric that compares every forecast against two specific reference points. The first is a "hindcast," which is a perfect look-back. Imagine if you could see the future and then write down exactly what happened, accounting for all the noise and errors in the data collection. This represents the absolute best anyone could possibly do. The second reference point is a "baseline," a simple guess that a computer could make without any complex modeling. By measuring how much better a real forecast is than a simple guess, and how close it gets to the perfect look-back, the researchers could finally see if the field was truly advancing.
The team applied this new measuring stick to more than 2.5 million forecasts generated over ten years. They looked at predictions for influenza-like illness, hospitalizations, and deaths, as well as similar targets for COVID-19. They tested these forecasts against three different types of simple baselines: one that just looked at the long-term history of the disease, one that assumed the next week would look exactly like the last week, and one that followed the recent trend of the data. The results were clear for the seasonal flu. The collective effort of all the forecasting teams, known as an ensemble, consistently outperformed the simple guesses. More importantly, the skill of these forecasts improved steadily over time. A forecast made four weeks in advance for the most recent flu season was more accurate than a forecast made just one week in advance during the very first year of the challenge. This suggests that as teams have had more time to learn, more data to study, and better tools to use, they have genuinely gotten better at seeing the future of the flu.
The story for COVID-19 was more complicated, reflecting the unique challenges of a new disease. In the early days of the pandemic, when there was almost no historical data to learn from, the forecasts were surprisingly good at predicting cases and deaths, beating the simple baselines. However, as the pandemic evolved and the rules of the game changed—such as when testing methods shifted or reporting systems were updated—the accuracy of the forecasts dipped. The models struggled to keep up with these sudden changes in how data was collected. While the forecasts for hospitalizations eventually showed signs of improvement, the predictions for daily cases and deaths did not show the same clear upward trend. This highlights a key lesson: forecasting works best when the patterns of the disease are stable and the data is consistent. When the ground shifts beneath the models, even the smartest algorithms struggle to find their footing.
The researchers found that the most successful forecasts were not just the result of better math or faster computers, but of better data and sustained collaboration. The flu forecasts benefited from decades of consistent reporting, allowing models to learn the rhythm of the seasons. In contrast, the struggles with COVID-19 case predictions were often tied to changes in how cases were counted, which introduced confusion that no model could easily untangle. The study also revealed that while the best collective forecasts were very strong, many individual teams still struggled to beat the simplest of baselines. This suggests that while the field as a whole is moving forward, there is still a wide gap between the best models and the average ones. The new skill score framework provides a way to see this progress clearly, separating the difficulty of the task from the quality of the prediction. It shows that infectious disease forecasting is a maturing science, one that is learning to navigate the messy reality of human disease, but it also reminds us that some of the most unpredictable moments in public health remain just out of reach.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.