Insights from the second season of collaborative influenza forecasting in Italy with updated targets incorporating virological information

The second season of Italy's Influcast collaborative forecasting hub demonstrated that ensemble models integrating virological data into influenza-like illness targets significantly outperform baseline and individual models, highlighting the value of combining syndromic and virological surveillance for more reliable epidemic forecasting.

The Gist

Imagine you are trying to predict the weather for the next few weeks. Usually, you look at the clouds, the wind, and the temperature. But what if you could also peek inside the clouds to see exactly how much rain is actually forming versus just mist? You would get a much clearer picture of whether to bring an umbrella or a raincoat.

This paper is about doing exactly that, but for flu season in Italy.

Here is the story of their "second season" of forecasting, broken down into simple concepts:

1. The Old Way: Guessing by Symptoms

For years, public health officials have tracked the flu by counting people who feel sick. They call this ILI (Influenza-Like Illness).

• The Problem: If you have a fever and a cough, you might have the flu. But you could also have a cold, RSV, or even just a bad allergy.
• The Analogy: Imagine trying to count how many Red Apples are in a fruit basket, but you are only allowed to count anything that is Red and Round. You might accidentally count a red tomato or a red grape. Your count is "noisy" and confusing because it mixes different fruits together. This makes it hard to predict exactly when the "Red Apple" season will peak.

2. The New Way: The "Flu-Only" Filter

In this study, the researchers (a team of scientists from universities and hospitals) tried a new trick. They took the "Red and Round" count (the sick people) and filtered it using lab test results.

• They asked: "Of all the people who are sick, how many actually tested positive for Flu A? How many for Flu B?"
• They created a new target called ILI+. This is like taking that fruit basket and only counting the items that are proven to be Red Apples.
• The Result: The signal became much cleaner. It was no longer a mix of "flu + cold + allergy"; it was just "flu."

3. The Team Race: The "Super-Team" vs. The Solo Players

The researchers set up a competition (a "forecasting hub") where six different computer models tried to predict the flu.

• The Solo Players: These were individual models, some using complex physics-like equations (mechanistic) and others using pure statistics.
• The Super-Team (The Ensemble): They took all six models, asked them for their predictions, and averaged them together.
• The Baseline: They also had a "naive" model that just guessed, "Next week will be exactly like this week." (Like saying, "If it rained today, it will rain tomorrow.")

4. What Happened?

The results were a big win for the new method:

• The Filter Worked: When the models tried to predict the "Noisy" old data (ILI), they struggled. But when they switched to the "Clean" new data (ILI+), they got much better. It was like the models finally stopped trying to guess about the tomatoes and grapes and focused entirely on the apples.
• The Team Won: The "Super-Team" (the ensemble) was the most reliable predictor. It consistently beat the "naive" guesser and most of the solo players.
• Better Calmness: The new method didn't just predict better; it was also more honest about its uncertainty. It knew when it was confident and when it wasn't, whereas the old method often got overconfident and made mistakes.

5. Why Does This Matter?

Think of public health officials as the firefighters of the community.

• Before: They were getting a smoke alarm that went off for every smell of smoke, whether it was a real fire, burnt toast, or a candle. They didn't know if they needed a fire truck or just a fan.
• Now: With the new "ILI+" method, the alarm tells them specifically, "It's a fire, and it's spreading fast."

This allows officials to:

• Send help exactly where it's needed.
• Tell the public, "The flu is peaking now, get your vaccine," with much more confidence.
• Stop wasting resources on false alarms.

The Bottom Line

This paper proves that if you want to predict a disease outbreak, you shouldn't just look at how many people are sneezing. You need to know what they are sneezing from. By mixing the "sick person" data with "lab test" data, the forecasters in Italy built a much sharper, more accurate crystal ball for the flu. It's a small change in the data, but it makes a huge difference in saving lives.

Technical Summary

1. Problem Statement

Seasonal influenza forecasting is critical for public health planning, yet traditional forecasting targets often rely on Influenza-like Illness (ILI) incidence. ILI is a syndromic indicator (based on symptoms like fever and cough) rather than pathogen-specific confirmation.

• Limitations of ILI: ILI aggregates multiple respiratory pathogens (e.g., RSV, rhinovirus, SARS-CoV-2, and influenza A/B). This "noise" obscures distinct epidemiological dynamics, making it difficult to disentangle influenza transmission from other co-circulating viruses.
• The Gap: While virological surveillance data (lab-confirmed cases) has expanded, its integration into real-time collaborative forecasting hubs to refine target definitions remains under-explored. It is unclear if shifting from syndromic (ILI) to pathogen-specific (ILI+) targets improves the accuracy and reliability of ensemble forecasts.

2. Methodology

The study utilized Influcast, a multi-model collaborative forecasting hub operating in Italy, during the 2024/25 winter season.

A. Data and Targets

The study compared forecasts for three distinct targets:

1. Standard ILI: Weekly incidence of ILI cases per 1,000 patients (syndromic data only).
2. ILI+(Flu A): Weekly incidence of ILI cases attributable specifically to Influenza A.
3. ILI+(Flu B): Weekly incidence of ILI cases attributable specifically to Influenza B.

Construction of ILI+ Targets:
The ILI+ targets were constructed by integrating syndromic surveillance with virological data from the Italian National Institute of Health (RespiVirNet). The formula used was:
$$\text{ILI}^+(\text{Flu X}) = \text{ILI} \times \frac{\text{positives}_{\text{Flu X}}}{\text{total positives}}$$
Where "positives" refers to laboratory-confirmed samples for the specific virus type, and "total positives" refers to all monitored respiratory pathogens.

B. Modeling and Forecasting

• Participants: Six independent modeling teams submitted probabilistic forecasts (1- to 4-week horizons).
• Model Types: The submissions included five mechanistic models (one metapopulation, four compartmental) and one statistical model.
• Ensemble: Individual forecasts were combined using a mean-across-quantiles approach (Vincent average method) to create an ensemble forecast.
• Baseline: A naive persistence model (assuming the next week's value equals the last observed value) served as the benchmark.
• Timeline: 18 forecasting rounds were analyzed from Week 48 of 2024 to Week 13 of 2025.

C. Evaluation Metrics

Performance was evaluated using:

1. Weighted Interval Score (WIS): A proper scoring rule combining sharpness (interval width) and calibration.
2. Absolute Error (AE): The absolute deviation of the median forecast from the observed value.
3. Prediction Coverage: The proportion of observed values falling within the 50% and 90% prediction intervals.
4. Relative Performance: Metrics were normalized against the baseline (ratio < 1 indicates improvement).

3. Key Contributions

1. Novel Target Definition: The paper operationalizes the integration of virological data into real-time forecasting targets (ILI+), moving beyond pure syndromic indicators.
2. Systematic Evaluation: It provides the first systematic comparison of ensemble and individual model performance on these refined targets versus standard ILI targets within a collaborative hub setting.
3. Methodological Validation: It validates that "cleaner" targets (isolating specific pathogens) lead to measurable improvements in forecast accuracy and calibration, even with a limited number of participating models.

4. Key Results

A. Ensemble Performance on ILI+ Targets

• The Ensemble model consistently ranked among the top performers for both ILI+(Flu A) and ILI+(Flu B).
• WIS and AE: The ensemble achieved significantly lower (better) relative WIS and AE ratios compared to the baseline.
  • For ILI+(Flu A): Ensemble WIS ratio was 0.38 (2nd best), AE ratio 0.40 (2nd best).
  • For ILI+(Flu B): Ensemble WIS ratio was 0.41 (2nd best), AE ratio 0.45 (2nd best).
• Stability: The ensemble demonstrated superior stability, appearing in the top half of rankings in >80% of rounds for Flu A and >90% for Flu B.

B. Comparison: ILI vs. ILI+

• Superior Accuracy: Integrating virological data significantly improved predictive accuracy. The median WIS ratio between the Ensemble and Baseline was 0.82 for standard ILI, compared to 0.36 for ILI+(Flu A) and 0.47 for ILI+(Flu B).
• Calibration: The ensemble forecasts for ILI+ targets showed better calibration (prediction coverage closer to nominal levels).
  • Average 90% coverage: 0.46 for ILI vs. 0.60 (Flu A) and 0.64 (Flu B) for ILI+.
  • Standard ILI forecasts were systematically overconfident (coverage < 90%) leading up to the epidemic peak, whereas ILI+ forecasts were more reliable.
• Horizon Dependency: The performance gains of ILI+ over ILI were most pronounced at shorter horizons (1–2 weeks) and diminished slightly as the forecast horizon increased, though ILI+ remained superior.

C. Individual Model Performance

• Most individual models also benefited from the refined targets, showing lower WIS and AE ratios for ILI+ compared to standard ILI.
• Flu A vs. Flu B: Performance gains were generally larger for Flu A, likely due to higher circulation and earlier peaks compared to Flu B during the 2024/25 season.

5. Significance and Implications

• Public Health Utility: By isolating specific pathogens, ILI+ targets provide clearer insights into the dynamics of co-circulating viruses. This allows for more precise risk communication and targeted interventions (e.g., distinguishing influenza surges from RSV surges).
• Forecasting Best Practices: The study confirms that refining forecasting targets using available virological data is a high-value strategy for improving short-term epidemic forecasts.
• Future Directions: The authors suggest extending this framework to other respiratory pathogens (RSV, SARS-CoV-2) and exploring multi-pathogen forecasting approaches where aggregate ILI signals are reconstructed from pathogen-specific predictions.

Conclusion: The integration of virological surveillance data into forecasting targets (ILI+) significantly enhances the accuracy, reliability, and calibration of ensemble influenza forecasts in Italy, offering a robust framework for future real-time epidemic monitoring.