Methods of adjustment for public health and social measures in post-licensure vaccine studies in children in sub-Saharan Africa: a systematic review

This systematic review of 64 post-licensure vaccine studies in sub-Saharan Africa reveals that none accounted for concurrent public health and social measures, highlighting a critical gap in study design that may compromise the interpretability of vaccine effectiveness estimates, particularly for pathogens like pneumococcus where results showed substantial heterogeneity.

The Gist

Imagine you are a detective trying to figure out if a new, magical shield (a vaccine) is the reason a village is suddenly seeing fewer sick children.

You look at the data and see: "Hey! Since we gave out the shields, the number of sick kids dropped by 50%!"

But here's the problem: Around the same time the shields were handed out, the village also got a new water pump, better food supplies, and a team of doctors started visiting more often. These are the Public Health and Social Measures (PHSMs).

The big question is: Did the shield save the day, or was it the new water pump and food? Or maybe it was a mix of both?

This paper is a massive investigation into 64 different "detective cases" (studies) from Sub-Saharan Africa between 2000 and 2019. The authors wanted to know: Did these detectives account for the other changes happening in the village, or did they just blame the shield for everything?

The Big Discovery: The "Blind Spot"

The authors found a shocking "blind spot" in almost every single study.

None of the 64 studies checked to see if the water pump, the food, or the doctors were helping. They only looked at the vaccine.

It's like trying to judge how good a new car engine is, but you forget to mention that you also replaced the tires, fixed the brakes, and put premium gas in the tank. You might say, "The engine is amazing!" but you don't know if it's actually the engine or the gas that made the car go faster.

The Different Types of "Detectives" (Study Designs)

The paper looked at how these studies were done:

1. The "Snapshot" Detectives (Case-Control): They looked at sick kids and healthy kids and asked, "Did you have the shield?" This was the most common method.
   • The Flaw: They didn't ask, "Did you also get better food?"
2. The "Trend Watchers" (Ecological Studies): They looked at the whole village's health records over time. "Before the shield, 100 kids were sick. After, only 50."
   • The Flaw: They couldn't tell if the drop was because of the shield or because the village got a new water treatment plant at the exact same time.
3. The "Long-Term Observers" (Cohort Studies): They followed kids over time.
   • The Flaw: Even these careful observers forgot to write down if the kids' families got better nutrition or access to clean water.

The Results: Did it Matter?

The authors asked: "If we ignore the water pump and food, does it change our answer about the shield?"

• For Rotavirus (a stomach bug): Surprisingly, the answer was mostly no. Even without checking for food or water, the studies agreed that the shield worked very well. The "magic shield" seemed to be doing the heavy lifting, regardless of other changes.
• For Pneumococcus (a lung infection): The answer was yes, it mattered a lot. The results were all over the place. Some studies said the shield was great; others said it was just okay. The authors think this is because the shield's success depends heavily on the "other stuff" (like whether the kids had HIV or how clean their air was). If you don't measure those other factors, you get a confusing mess of results.
• For other vaccines (Measles, Flu, etc.): The results varied. Some were clear winners, others were shaky.

The "Why" and The "What Now"

Why didn't they check?
In many parts of Africa, keeping track of data is hard. It's like trying to count every single raindrop in a storm while also trying to measure the wind speed. The systems to record "did this family get better food?" or "did this village get a new well?" often don't exist or aren't connected to the vaccine records.

What should we do?
The authors are calling for a change in how we do detective work:

1. Bring a Notebook: Future studies need to write down everything that changes. Did the village get new nets for malaria? Did a new clinic open? Did food prices drop?
2. Better Tools: We need to use study designs that can handle these messy, real-world changes. Imagine a study where you give the shield to one group of villages and not another, but you make sure both groups get the same new water pumps. That way, you know for sure the shield is the hero.
3. Fix the Systems: Governments need to upgrade their record-keeping. If a hospital records a sick child, they should also be able to check: "Was this child malnourished? Did they have clean water?"

The Bottom Line

Vaccines are incredible tools, and they are saving lives. But to know exactly how much they are saving, and to make sure we aren't wasting money on the wrong things, we need to stop looking at the vaccine in isolation.

We need to look at the whole picture. If we don't, we might give all the credit to the shield when it was actually the water pump, or worse, we might think the shield isn't working when it actually is, just because we didn't account for the other changes happening around it.

In short: To know if the vaccine is the hero, we have to stop ignoring the rest of the cast of characters in the story.

Technical Summary

1. Problem Statement

Post-licensure vaccine effectiveness (VE) and impact studies are critical for understanding how vaccines perform in real-world routine program conditions. In Sub-Saharan Africa (SSA), vaccine introductions often occur concurrently with other public health and social measures (PHSMs), such as improvements in water, sanitation, and hygiene (WASH), nutritional interventions, malaria control, and HIV prevention programs.

The Core Issue: These concurrent PHSMs can independently reduce disease burden and transmission. If studies fail to measure or adjust for these factors, the observed reduction in disease may be incorrectly attributed solely to the vaccine (confounding), leading to biased effectiveness estimates. The review addresses a critical methodological gap: to what extent do existing post-licensure studies in SSA account for these concurrent interventions?

2. Methodology

The authors conducted a systematic review adhering to PRISMA guidelines and registered with PROSPERO (CRD42023436851).

• Search Strategy: A comprehensive search of PubMed Central, EMBASE, MEDLINE, and CINAHL was performed for peer-reviewed articles published between January 2000 and December 2019. The search was restricted to English-language studies involving children under five in SSA.
• Inclusion Criteria:
  • Observational study designs (case-control, cohort, time-series, ecological, cluster-randomized).
  • Estimation of vaccine effectiveness or population-level impact.
  • Focus on children <5 years in SSA.
• Exclusion Criteria: Clinical trials, pre-prints, modeling studies without observational data, cost-effectiveness studies, and studies not reporting effect measures.
• Data Extraction & Quality Assessment: Two independent reviewers screened studies, extracted data (including study design, vaccine, outcomes, and PHSM adjustments), and assessed methodological quality using Joanna Briggs Institute (JBI) tools.
• Analysis: A narrative synthesis was performed. Additionally, a meta-analysis was conducted for rotavirus and pneumococcal vaccines using random-effects models to assess heterogeneity in individual-level effectiveness estimates across comparable study designs.

3. Key Contributions

• First Comprehensive Assessment: This is the first systematic review to specifically evaluate the extent to which post-licensure vaccine studies in SSA account for concurrent PHSMs in their design or analysis.
• Identification of a Universal Gap: The review definitively establishes that none of the 64 included studies collected, reported, or adjusted for PHSMs (e.g., nutrition, WASH, healthcare access).
• Pathogen-Specific Variability: The paper demonstrates that the consequences of this omission are not uniform; some vaccine estimates appear robust to unmeasured context, while others are highly sensitive to it.
• Methodological Recommendations: It provides specific, design-oriented recommendations for future studies to better capture and adjust for time-varying contextual factors.

4. Key Results

Study Characteristics:

• Total Studies: 64 studies met inclusion criteria, covering 8 vaccine-preventable diseases.
• Vaccines: Rotavirus (48.4%) and Pneumococcal conjugate (14.1%) were the most studied.
• Study Designs: Case-control (45.3%) and Ecological (37.5%) were dominant. Cohort (6.3%) and Time-series (10.9%) were less common.
• Geography: Southern Africa (45.3%) and West Africa (29.7%) were overrepresented; Central and Northwest Africa were underrepresented.
• Funding: Primarily Non-Profit Organizations (54.7%) and Public institutions.

Findings on PHSM Adjustment:

• Zero Adjustment: 0% of the 64 studies reported collecting data on or adjusting for PHSMs.
• Design Limitations:
  • Case-control: Relied on retrospective data; assumed cases/controls had comparable exposure to PHSMs, an assumption rarely tested.
  • Ecological: Used aggregate surveillance data which lacks individual-level PHSM data, making it impossible to disentangle vaccine effects from concurrent interventions.
  • Cohort/Time-series: Despite potential for longitudinal adjustment, none incorporated time-varying PHSM data due to lack of routine data sources.

Vaccine Effectiveness & Heterogeneity:

• Rotavirus: Effectiveness estimates from comparable individual-level designs were consistent across settings (Pooled OR 0.42; $I^2 = 0\%$). This suggests rotavirus VE estimates may be robust to unmeasured PHSMs when study designs are aligned.
• Pneumococcal (PCV): Estimates showed substantial heterogeneity. Variation was linked to outcome definitions (IPD vs. clinical pneumonia), HIV status, and setting, suggesting these estimates are highly sensitive to contextual factors.
• Other Vaccines:
  • Measles: Consistently high protection (74–90%).
  • Influenza: Highly variable (−14% to 67%), driven by seasonal strain mismatch and context.
  • Hib: Consistently high effectiveness (83–99%).
  • Meningococcal: Conjugate vaccines showed high impact (~89%), while polysaccharide vaccines showed no significant impact.

5. Significance and Implications

Scientific Significance:
The review highlights that the lack of PHSM adjustment is a systemic methodological flaw in SSA vaccine research. While some estimates (like rotavirus) appear stable, others (like pneumococcal and influenza) are likely confounded by unmeasured concurrent interventions. This limits the ability to compare effectiveness across different countries or time periods accurately.

Policy and Practice Implications:

• Resource Allocation: Failure to adjust for PHSMs may lead to overestimating vaccine impact, potentially causing health authorities to under-invest in complementary interventions (e.g., WASH, nutrition) that are synergistic with vaccination. Conversely, underestimation could lead to deprioritizing effective vaccines.
• Data Systems: There is an urgent need to strengthen routine health information systems (e.g., DHIS2) in LMICs to capture PHSM indicators (nutrition status, WASH access, bed net usage) alongside vaccination data.
• Study Design Recommendations:
  • Case-Control: Should collect individual-level data on nutrition and WASH for matching or stratification.
  • Ecological/Time-Series: Should link surveillance data with demographic and health survey data to model concurrent interventions.
  • Cluster Randomized Trials: Where feasible, these designs (e.g., RTS,S malaria vaccine pilot) can balance unmeasured contextual factors across arms, providing more robust population-level estimates.

Conclusion:
The authors conclude that future post-licensure evaluations in SSA must prioritize the systematic measurement of PHSMs. Integrating these factors into study design and analysis is essential for generating interpretable evidence to guide immunization policy and maximize public health gains in rapidly changing environments.