📊 statistics

A simple and powerful test of vaccine waning

This paper proposes a powerful new statistical test to assess individual-level vaccine waning that overcomes the limitations of existing methods by requiring fewer assumptions and offering greater power, successfully detecting waning in BNT162b2 COVID-19 vaccine data where prior analyses failed.

Original authors: Gellért Perényi, Matias Janvin, Mats J. Stensrud

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gellért Perényi, Matias Janvin, Mats J. Stensrud

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Survivor" Trap

Imagine you are testing a new umbrella to see if it keeps you dry in the rain.

Month 1: You give umbrellas to 100 people and no umbrellas to 100 others. It rains hard. The umbrella group gets wet 20% of the time; the no-umbrella group gets wet 80% of the time. The umbrella is working great!
Month 3: You check the data again. But here is the catch: You can only check the people who didn't get wet in Month 1.

In the "no umbrella" group, the people who got wet in Month 1 are gone from your list. The only people left are the ones who are naturally tough and wouldn't get wet even without an umbrella (maybe they have a natural raincoat). In the "umbrella" group, you still have a mix of people who need the umbrella and those who are naturally tough.

If you compare the two groups now, the "no umbrella" group looks surprisingly dry because the weak ones are already gone. The "umbrella" group still has some people getting wet. It might look like the umbrella stopped working, even if it's still perfectly strong.

The Paper's Point: Traditional ways of checking if a vaccine "wanes" (gets weaker over time) often fall into this trap. They compare the wrong groups of people, leading to confusing or false conclusions about whether the vaccine is actually losing its power.

The Proposed Solution: A "Time-Travel" Test

The authors propose a new, simpler test to see if a vaccine's protection is truly fading at the individual level.

Think of it like a Time-Travel Challenge.
Imagine we could freeze time and run a special experiment where we take a group of people, isolate them in a bubble for a month, and then expose them to the virus. Then, we take another group, isolate them for three months, and expose them to the virus.

If the vaccine works the same way at Month 1 and Month 3, the "failure rate" (getting sick) should be identical in both groups, provided we are comparing people who were equally likely to get sick before the experiment started.

The authors realized we can't actually do this time-travel experiment (it's unethical to force people to get sick). However, they found a mathematical "loophole."

The "Incidence Ratio" Trick

Instead of trying to measure the exact strength of the vaccine (which is hard), they ask a simpler question: "Is the ratio of sickness the same in both groups at both times?"

Time 1: How many vaccinated people got sick vs. how many unvaccinated people got sick?
Time 2: How many vaccinated people got sick vs. how many unvaccinated people got sick?

The Rule: If the vaccine is not waning (it's just as strong at Month 3 as it was at Month 1), the ratio of sickness between the two groups should stay exactly the same.

If the ratio changes significantly, it's a strong signal that the vaccine's protection has changed for the individuals.

Why This is Better

It's a "Sharp" Test: The authors call this a "sharp null hypothesis." It's like a light switch: either the vaccine protection is constant, or it isn't. They don't need to guess the exact amount of waning; they just need to prove the protection isn't staying the same.
It Avoids the "Survivor" Trap: By looking at the ratio of new infections in specific time windows, this method avoids the bias of comparing "survivors" who are naturally different from each other.
It Works with Old Data: You don't need a new, fancy experiment. You can take data from standard vaccine trials that have already been published, look at the numbers of sick people at different times, and run this test.

Real-World Examples from the Paper

The authors tested their method on two real studies:

The COVID-19 Vaccine (BNT162b2):
- Old Analysis: Previous studies looked at the data and couldn't say for sure if the vaccine was waning because the numbers were too fuzzy.
- New Test: The authors applied their new test. It gave a clear "Yes." The ratio of sickness changed significantly, proving that the vaccine's protection did wane over time.
The H. pylori Vaccine (Stomach Bacteria):
- Old Analysis: Studies showed the vaccine was working well at Month 8 and Month 12.
- New Test: The authors applied their test. The ratio of sickness stayed the same. The test said "No waning." This matched what scientists expected based on biology (the immune response didn't drop).

The Bottom Line

The paper introduces a simple, powerful math trick to cut through the noise of vaccine data. Instead of getting confused by which people are left in the study, it compares the rate of new infections between vaccinated and unvaccinated groups at different times.

If that rate ratio shifts, the vaccine is likely waning. If it stays steady, the vaccine is holding strong. This allows scientists to use existing data to make clearer, more confident decisions about when booster shots might be needed.

Title: A Simple and Powerful Test of Vaccine Waning

Authors: Gellért Perényi, Matias Janvin, and Mats J. Stensrud

Problem Statement

Determining whether vaccine efficacy (VE) wanes over time is critical for public health decision-making. However, quantifying waning is methodologically subtle. Conventional approaches, which compare VE estimates at different time points (e.g., VE at 1 month vs. VE at 3 months), are subject to selection bias due to the "depletion of susceptibles." As time progresses, the population remaining at risk in the placebo group differs from the initial population because those most susceptible to infection have already experienced the outcome. Consequently, a decline in observed VE over time may reflect this changing population composition rather than a true decline in the biological protection offered by the vaccine.

While recent work (Janvin et al., 2024) proposed causal estimands (challenge effects) to quantify waning under weaker assumptions, these estimands are often only partially identifiable in standard randomized controlled trials (RCTs). The resulting bounds are frequently too wide to definitively claim the presence or absence of waning. Furthermore, standard time-to-event analyses using hazard ratios (HRs) are unsuitable for testing waning because HRs are also affected by the depletion of susceptibles, leading to biased interpretations even when the underlying vaccine effect is constant.

Methodology

The authors propose a formal hypothesis test to assess whether a treatment effect is constant over time at the individual level, grounded in a causal framework.

1. Target Estimand and Hypothesis:
The authors define a "sharp null hypothesis" of no waning (Hypothesis 1). This hypothesis posits that for any individual, the potential outcome upon exposure is identical regardless of whether the exposure occurs at time 1 or time 2 (assuming isolation at time 1). Formally:
$\Delta Y^{a, e_1=1}_1 = \Delta Y^{a, e_1=0, e_2=1}_2 \quad \text{w.p.1} \quad \forall a \in \{0, 1\}$
This implies that individual susceptibility is constant across intervals and independent of exposure timing. The authors argue that under reasonable assumptions (specifically, that the placebo effect does not change over time and there is no perfect cancellation of positive and negative waning), this individual-level sharp null is equivalent to the population-level notion of no waning ( $VE^{challenge}_1 = VE^{challenge}_2$ ).

2. Key Assumptions:
The test relies on four primary assumptions, which are plausible in blinded RCTs:

Exposure Necessity: An outcome cannot occur without exposure.
Exposure Exclusion Restriction: Exposure at time 1 does not affect the outcome at time 2 except through survival (no carry-over effects like induced immunity from asymptomatic infection).
No Treatment Effect on Exposure: In a blinded trial, treatment assignment does not influence the probability of seeking exposure at time 1 or time 2 (conditional on survival).
Exposure Exchangeability: Exposure patterns are exchangeable across treatment arms, conditional on survival and baseline characteristics.

3. The Test Statistic:
Under the null hypothesis and the stated assumptions, the authors derive a simple equality: the ratio of incidence rates (IR) in the first interval must equal the ratio of incidence rates in the second interval.
$IR_1 = \frac{E[\Delta Y_1 | A=1]}{E[\Delta Y_1 | A=0]} = \frac{E[\Delta Y_2 | A=1]}{E[\Delta Y_2 | A=0]} = IR_2$
The test statistic is the ratio of these incidence rates ( $IR_1 / IR_2$ ). If the null hypothesis holds, this ratio should be 1. The authors provide three methods to compute confidence intervals for this ratio:

A direct $\delta$ -method applied to the ratio.
A conservative $\delta$ -method.
Non-parametric bootstrap (requiring individual-level data).

Crucially, the authors demonstrate that unlike hazard ratios, incidence ratios are not subject to the same selection bias regarding the depletion of susceptibles in this specific testing framework.

Key Contributions

A Novel Test for Waning: The paper introduces a sharp null hypothesis test that avoids the selection bias inherent in comparing VE estimates over time or using hazard ratios.
Power and Simplicity: The proposed test offers considerable power gains over existing partial identification approaches (Janvin et al., 2024). It can be implemented using only summary data (incidence counts) available from most published RCTs, without requiring individual-level data or complex modeling of unobserved heterogeneity.
Theoretical Clarification: The work clarifies the distinction between population-level waning and individual-level waning, showing that under plausible conditions (no perfect cancellation of effects), rejecting the sharp null implies population-level waning.
Bounds Improvement: Beyond the test, the authors provide new results that bound the waning effect, offering a strategy to improve the bounds derived by Janvin et al. (2024) when additional data (e.g., contact tracing) is available.

Results

The authors illustrate the method through real and simulated examples:

BNT162b2 (COVID-19) Reanalysis: Reanalyzing data from the BNT162b2 trial (Thomas et al., 2021), the authors found that while the previous analysis by Janvin et al. (2024) could not reject the null of no waning due to wide bounds, the new test rejected the null hypothesis ( $P=0.01$ ). The estimated ratio of incidence ratios was 0.48, providing evidence of waning.
Helicobacter pylori Vaccine: In a reanalysis of a trial by Zeng et al. (2015), the test did not reject the null hypothesis ( $P=0.97$ ). This result aligned with biological explanations (stable antibody levels) and the original trial's findings of stable efficacy, demonstrating the test's ability to correctly identify the absence of waning.
Simulations: Simulation studies confirmed that the test maintains the correct Type I error rate under the null hypothesis and exhibits higher power than tests based on hazard ratios or conservative bounds, particularly when exposure levels vary.

Significance and Claims

The authors claim that their method provides a "simple and powerful" tool for assessing vaccine waning that is valid under interpretable assumptions in conventional vaccine trials.

Validity: The test is valid under assumptions that are standard for blinded RCTs, avoiding the "unreasonable assumptions" required by classical approaches.
Utility: It allows researchers to detect waning in existing trials where previous methods failed to provide definitive answers.
Limitations: The authors modestly note that while the test is powerful for detecting the existence of waning, it is generally uninformative about the precise magnitude of waning without further assumptions or data. They suggest that if waning is confirmed, further steps (e.g., follow-up trials or reanalysis with additional assumptions) are needed to quantify the extent of the decline.
Scope: The paper emphasizes that the test assesses the existence of waning. If the test rejects the null, it indicates a change in the population challenge effect, though the specific mechanism (e.g., "leaky" vs. "all-or-nothing" vaccine effects) may require further investigation.

In summary, the paper offers a rigorous, assumption-lean statistical framework to distinguish true vaccine waning from artifacts caused by the depletion of susceptibles, enabling more reliable inferences from standard RCT data.