Validation of methods for forecasting the frequency of non-vaccine serotypes after introduction or switch of a pneumococcal conjugate vaccine

This study developed and validated an accuracy-weighted ensemble model that effectively forecasts the frequency of non-vaccine pneumococcal serotypes following vaccine introduction or switching, thereby providing a valuable tool for guiding future vaccine impact assessments and formulation updates.

The Gist

Imagine the human nose as a bustling city square. In this square, there are many different "families" of bacteria called Streptococcus pneumoniae. Each family wears a unique hat (a serotype) that identifies them. Some of these families are troublemakers that cause serious sickness (Invasive Pneumococcal Disease, or IPD), while others are just hanging out without causing harm.

For a long time, the city was dominated by a few specific troublemaker families. To stop them, scientists built a "security fence" called a vaccine. The first fence, PCV7, blocked the 7 most dangerous families. When these families were kicked out of the square, a problem arose: the empty space they left behind didn't stay empty. Other, previously harmless families (the Non-Vaccine Serotypes or NVTs) rushed in to fill the gap, wearing their own unique hats. This is called "serotype replacement."

Now, scientists are building bigger fences (PCV13, PCV15, PCV20) to block more families. But they face a tricky question: When we block more families, which new troublemakers will rush in to take their place?

This paper is like a team of meteorologists trying to predict the weather of this bacterial city. They wanted to build a crystal ball to forecast exactly which bacterial families would become dominant after a new vaccine is introduced.

The Three Weather Models

The researchers didn't rely on just one crystal ball. Instead, they built three different models, each with a different way of thinking about how the bacterial city works:

1. The "Proportionate" Model (The Fair Share Approach):

   • The Idea: Imagine the bacterial city square has a fixed amount of space. If the vaccine removes 50% of the space occupied by the old troublemakers, the new families will simply split that empty space evenly among themselves, proportional to how big they were before.
   • Analogy: If you remove the top 3 players from a sports league, the remaining teams just split the wins based on their previous performance.

2. The "Ranking" Model (The Ladder Approach):

   • The Idea: This model assumes that the order of the bacteria matters more than their exact size. The most popular bacteria will always be #1, the next most popular will be #2, and so on. When the vaccine removes the top few, the bacteria below simply slide up the ladder.
   • Analogy: Think of a waiting line at a coffee shop. If the first three people leave, the person who was #4 instantly becomes the new #1, #5 becomes #2, etc. The "Ranking" model predicts who moves up the line.

3. The "NFDS-lite" Model (The Family Resemblance Approach):

   • The Idea: This model looks at genetics. It assumes that bacteria that are "cousins" (genetically similar) to the ones being vaccinated against are the most likely to take over. If you block a specific family, their genetic cousins are the ones most likely to sneak in and fill the void.
   • Analogy: If you ban a specific type of car in a city, the cars that look most like that banned model (same make, similar color) are the ones most likely to appear in the parking spots left empty.

The "Super-Model" (The Ensemble)

The researchers realized that no single crystal ball is perfect. Sometimes the "Fair Share" approach works best; other times, the "Family Resemblance" approach is more accurate.

So, they created an Ensemble Model. Think of this as a panel of expert judges. Instead of listening to just one judge, they listen to all three models, weigh their past performance, and combine their opinions into one final, highly accurate prediction.

What Did They Find?

• The Prediction Works: By looking at data from the US and other countries, they tested their models against real-world history. They found that their "Super-Model" was very good at predicting which bacterial families would rise to power after the vaccines were introduced.
• Timing Matters: The models were better at predicting the long-term results (several years after the vaccine) than the immediate aftermath. It takes time for the bacterial city to settle into a new normal.
• Age Matters: The models worked particularly well for the very young and the elderly, likely because the vaccines had a very strong effect on these groups, making the bacterial shifts easier to track.

Why Does This Matter?

Imagine you are the mayor of the bacterial city. You need to know which new troublemakers are coming so you can build the right fence before they cause an epidemic.

This paper gives public health officials a powerful tool. Instead of guessing which bacteria will take over when we switch from a 13-valent vaccine to a 20-valent vaccine, they can use this "Super-Model" to forecast the outcome. This helps them decide:

• Which bacteria should be included in the next generation of vaccines.
• How effective the current vaccines will be in the long run.
• How to prepare hospitals for potential outbreaks of new bacterial strains.

In short, the authors built a sophisticated navigation system for the bacterial world, helping us stay one step ahead of the bacteria that try to fill the gaps left by our vaccines.

Technical Summary

1. Problem Statement

Pneumococcal conjugate vaccines (PCVs) have significantly reduced invasive pneumococcal disease (IPD) caused by vaccine-targeted serotypes (VTs). However, a major public health challenge is serotype replacement, where non-vaccine serotypes (NVTs) expand to fill the ecological niche vacated by VTs, potentially diminishing the overall public health impact of vaccination programs.

As the field moves from lower-valency vaccines (e.g., PCV7, PCV13) to higher-valency formulations (e.g., PCV15, PCV20) and next-generation vaccines (24–30+ valents), there is a critical need to accurately forecast NVT dynamics. Existing models often:

• Focus primarily on initial vaccine introduction rather than switches between vaccines.
• Aggregate serotypes into broad groups, failing to capture individual serotype behavior.
• Rely on carriage data that may not be available during the forecasting phase.
• Lack robust validation across different post-vaccine time periods and age groups.

2. Methodology

The authors developed and validated an ensemble forecasting framework designed to predict serotype-specific IPD frequencies following PCV introduction or switching.

Data Sources

• Primary Data: Age- and serotype-specific IPD case counts from the US CDC's Active Bacterial Core surveillance (ABCs) covering 1998–2019.
• Augmentation: To address low case counts and zero-inflation in the baseline period, data was augmented with IPD rates from countries that switched from PCV7 to PCV13 (Australia, UK, Israel, The Gambia).
• Invasiveness Data: Serotype-specific invasiveness (IPD cases per carrier per year) was derived from literature to infer carriage prevalence from IPD incidence.

Modeling Framework

The framework consists of three distinct statistical models, each making different assumptions about how NVTs expand, combined into a weighted Ensemble model.

1. Proportionate Model: Assumes a constant multiplier applied to pre-vaccine NVT prevalence. The magnitude of expansion is determined by the reduction in VTs (the "niche gap") based on relative vaccine efficacy against carriage.
2. Ranking Model: Assumes that while specific serotypes change, the relative rank order of serotype frequencies remains somewhat stable. It predicts expansion based on the shift in rank order as VTs are removed.
3. NFDS-lite (Negative Frequency-Dependent Selection-lite): Based on the biological principle that serotypes with similar capsular genes compete for the same niche. This model assumes the frequency of specific capsular genes remains stable, predicting that NVTs genetically related to VTs will expand the most.

Ensemble Construction

The three models were combined using an accuracy-weighted ensemble. Weights were assigned dynamically based on past model performance to leverage the strengths of each component model.

Validation Metrics

• Time Periods: Forecasts were generated for early/late post-PCV7 (2000–2009) and early/late post-PCV13 (2010–2019) periods.
• Accuracy: Evaluated using Normalized Root Mean Square Error (NRMSE), Pearson correlation, and Spearman rank correlation between predicted and observed IPD cases.
• Uncertainty: Propagated through 10,000 Monte Carlo simulations sampling from distributions of invasiveness and baseline incidence.

3. Key Contributions

• Switch-Focused Forecasting: Unlike previous studies focusing on initial introduction, this framework specifically addresses the complex dynamics of switching from one PCV to another (e.g., PCV7 to PCV13).
• Ensemble Approach: Demonstrates that combining models with distinct biological and statistical assumptions (Proportionate, Ranking, NFDS) yields more robust predictions than any single model.
• Data Augmentation: Developed a method to infer robust baseline carriage prevalence for rare serotypes by integrating international surveillance data, overcoming the limitation of sparse US baseline data for low-frequency serotypes.
• Granular Validation: Provided validation across five distinct age groups and multiple post-vaccine time windows, identifying where models succeed and fail.

4. Key Results

• Overall Accuracy: The Ensemble model successfully matched observed IPD patterns, with the highest accuracy observed during the post-PCV13 periods compared to post-PCV7 periods.
• Model Performance by Period:
  • Early Post-PCV7: The Proportionate model performed best (NRMSE 0.70–3.95), followed by NFDS-lite and Ranking.
  • Late Post-PCV7: The NFDS-lite model outperformed others (NRMSE 1.55–9.82), likely due to its ability to capture genetic relatedness and cross-protection effects that emerged over time.
  • Post-PCV13: All individual models performed comparably well, and the Ensemble model achieved the lowest error rates (NRMSE 0.31–3.09).
• Age Group Variations: Predictions were most accurate for infants (0–1y) and older adults (50+y), likely due to more substantial VT elimination and clearer NVT replacement signals in these groups. Accuracy was lower in middle-aged groups during the post-PCV7 era.
• Specific Serotype Challenges:
  • Serotypes 9A and 16A were poorly predicted during the post-PCV7 period (often overpredicted) but well-predicted in the post-PCV13 period. This suggests that cross-protection effects (e.g., PCV13 covering 9A) were better captured when using the post-PCV7 era as a baseline.
• Sensitivity Analysis: Prediction accuracy was highest when evaluating 4–6 years post-vaccine introduction. Reducing the assumed serotype replacement proportion in the Proportionate model significantly reduced accuracy for post-PCV7 predictions.

5. Significance and Implications

• Vaccine Policy: The framework provides a critical tool for public health officials to anticipate the impact of upcoming vaccine switches (e.g., to PCV15 or PCV20) and to monitor potential serotype replacement.
• Vaccine Formulation: By accurately forecasting which NVTs are likely to expand, the model can inform the selection of serotypes for future next-generation vaccines (24+ valents).
• Methodological Advancement: The study validates that an ensemble approach, which accounts for both ecological competition (NFDS) and simple proportional expansion, is superior for complex serotype dynamics.
• Limitations & Future Work: The authors note limitations regarding unaccounted cross-protection (e.g., serotype 6C) and the reliance on inferred rather than observed carriage data. Future work requires more longitudinal carriage studies to refine these models.

In conclusion, this paper establishes a validated, ensemble-based forecasting tool that significantly improves the ability to predict non-vaccine serotype dynamics following pneumococcal vaccine changes, offering a vital resource for guiding future vaccination strategies.