Evaluation of short-term multi-target respiratory forecasts over winter 2024-25 in England using sub-ensemble contribution analyses

This study evaluates the performance of UKHSA's operational multi-target respiratory forecasts for Influenza and COVID-19 hospital admissions in England during the 2024-25 winter, utilizing sub-ensemble simulations, generalized additive models, and Pareto analysis to quantify individual model contributions and reveal trade-offs between optimizing absolute count accuracy and trend direction.

The Gist

Imagine you are the captain of a massive ship navigating through a stormy winter sea. Your goal is to predict exactly where the waves will hit and how high they will get, so you can prepare your crew and passengers. In the real world, this "ship" is the National Health Service (NHS) in England, and the "storm" is the winter flu and COVID-19 season.

This paper is essentially a post-season report card for the team of weather forecasters (scientists) who tried to predict how many people would get sick and end up in hospitals during the winter of 2024–25.

Here is the story of their journey, broken down into simple parts:

1. The "Super-Team" Strategy (Ensembles)

Instead of relying on just one weather forecaster, the health officials created a "Super-Team." They combined predictions from about a dozen different computer models. Think of it like a panel of judges at a talent show. If one judge is too harsh and another is too lenient, the average of all their scores usually gives you the fairest result. This group of models working together is called an ensemble.

2. The Two Things They Tried to Predict

The team had to guess two different things, which is like trying to predict both the speed of a car and the direction it's turning at the same time:

• The Numbers: How many people will be admitted to the hospital? (The "Speed")
• The Trend: Will the number of sick people go up, down, or stay the same? (The "Direction")

3. The Big Challenge: The "Perfect" Mix

The researchers asked a tricky question: Is our current "Super-Team" actually the best possible team we could have built?

To find out, they ran a simulation. They took their library of models and tried out thousands of different combinations (sub-ensembles), like mixing and matching ingredients in a recipe. They wanted to see:

• Did adding a specific model help the team predict the numbers better?
• Did that same model help predict the direction better?

4. What They Discovered (The Plot Twist)

Here is the surprising part: There is no single "perfect" team.

• The Flu Team: When looking at the numbers of flu patients, the official team did a great job (47% better than random groups). But when looking at the direction (up or down), the official team actually performed worse than some smaller, specialized groups.
• The COVID Team: The official team struggled more here. Some smaller groups of models were much better at predicting the numbers, and even better at predicting the direction.

The Analogy: Imagine a sports team. You might have a striker who is amazing at scoring goals (predicting numbers) but terrible at passing (predicting trends). You might have a defender who is great at passing but can't score.

• If you put them all on the field together (the official ensemble), you get a balanced team.
• But if you only cared about scoring goals, you'd be better off with just the striker.
• If you only cared about passing, you'd want just the defender.

The paper found that optimizing for one thing often hurts the other. You can't always have the absolute best at both the "numbers" and the "direction" simultaneously.

5. Why This Matters

The authors used some fancy math (called GAMs and Pareto analysis) to map out these trade-offs. Think of it like a map that shows you the "efficient frontier"—the line where you can't improve one thing without making the other slightly worse.

The Takeaway:
The official forecasts were good and reliable (well-calibrated), but this study shows that by carefully selecting which models to include in the "Super-Team," we could potentially make even better predictions in the future. It's about knowing which "players" to put on the field depending on whether the priority is predicting the exact crowd size or just knowing if the crowd is growing or shrinking.

In short: This paper is a lesson in teamwork. It teaches us that while combining many voices usually helps, sometimes you need to be strategic about which voices you listen to, depending on what question you are trying to answer.

Technical Summary

1. Problem Statement

The paper addresses two critical challenges in epidemic forecasting, specifically regarding respiratory pathogens (Influenza and COVID-19) in England:

• Quantifying Individual Model Contributions: While ensemble forecasting (combining multiple models) is standard practice, it is difficult to isolate how specific component models affect overall ensemble performance, particularly when evaluating across multiple targets (e.g., hospital admission counts vs. trend direction) and varying spatial scales.
• Joint Optimization: Forecasting teams must balance the optimization of two distinct forecast types simultaneously:
  1. Absolute Counts: Predicting the specific number of hospital admissions.
  2. Trend Direction: Predicting the ordinal direction of the epidemic (e.g., rising, falling, or stable).
     Existing methods often struggle to identify sub-ensembles that are optimal for both metrics simultaneously, leading to potential trade-offs.

2. Methodology

The authors conducted a retrospective simulation of the Winter 2024-25 forecasting season in England, utilizing the operational modeling suite of the UK Health Security Agency (UKHSA).

• Data & Targets: Forecasts focused on Influenza and COVID-19 hospital admissions.
• Scoring Metrics:
  • Per Capita Weighted Interval Score (pcWIS): Used to evaluate the accuracy of absolute count forecasts.
  • Ranked Probability Score (RPS): Used to evaluate the accuracy of ordinal trend direction forecasts.
• Analytical Techniques:
  • Sub-ensemble Simulation: The authors generated various combinations of component models (sub-ensembles) to compare against the full operational ensemble.
  • Generalised Additive Models (GAMs): Applied to estimate the expected change in scoring metrics resulting from the inclusion or exclusion of specific models. This allowed for a granular assessment of individual model utility across different epidemic phases.
  • Pareto Analysis: Used to identify "Pareto-optimal" sub-ensembles—combinations where improving the score for one metric (e.g., counts) would inevitably degrade the score for the other (e.g., trend direction). This highlighted the inherent trade-offs in ensemble construction.

3. Key Contributions

• Novel Evaluation Framework: The study introduces a robust framework for decomposing ensemble performance using GAMs and Pareto analysis, moving beyond simple aggregate scoring to understand when and why specific models contribute value.
• Multi-Target Assessment: It explicitly addresses the complexity of optimizing for both absolute magnitude and directional trends simultaneously, a common but under-analyzed requirement in public health forecasting.
• Operational Retrospective Validation: It provides a rigorous, data-driven post-season evaluation of the UKHSA's Winter 2024-25 operational forecasts, offering actionable insights for future model selection.

4. Key Results

The analysis yielded distinct performance metrics for Influenza and COVID-19, revealing significant differences between operational ensembles and optimized sub-ensembles:

• Operational Performance:
  • Influenza: Achieved a pcWIS of $5.20 \times 10^{-7}$ and an RPS of 0.234.
  • COVID-19: Achieved a pcWIS of $3.98 \times 10^{-7}$ and an RPS of 0.171.
• Comparative Analysis (Operational vs. Sub-ensembles):
  • Influenza: The operational ensemble showed a 47% improvement in pcWIS (counts) compared to sub-ensembles. However, it performed 22% worse on average in RPS (trend direction) compared to optimal sub-ensembles.
  • COVID-19: The operational ensemble underperformed significantly compared to retrospective sub-ensembles, scoring 43% worse on pcWIS and 265% worse on RPS.
• Phase-Dependent Contributions: The GAM analysis confirmed that individual models influence ensemble performance differently depending on the epidemic phase (e.g., onset, peak, decline).
• Trade-off Identification: Pareto analysis demonstrated a clear trade-off: optimizing an ensemble for absolute count accuracy often comes at the expense of trend direction accuracy, and vice versa.

5. Significance and Interpretation

• Calibration: Despite the performance gaps in specific metrics compared to theoretical sub-ensembles, the operational UKHSA forecasts were well-calibrated against observations, indicating reliability for decision-making.
• Strategic Model Selection: The findings suggest that a "one-size-fits-all" ensemble is suboptimal. The study provides a methodological basis for dynamically selecting models based on the specific epidemic phase and the primary decision-making need (e.g., prioritizing trend direction vs. bed capacity planning).
• Future Forecasting: The integration of GAMs and Pareto analysis offers a roadmap for constructing future ensembles that can better navigate the trade-offs between different forecasting targets, ultimately improving the utility of forecasts for senior healthcare leaders managing winter hospital pressures.