Time-related biases in observational studies : an illustration with COVID-19 vaccination during immune checkpoint inhibitor therapy

This paper demonstrates how improper specification of study timelines in observational research can lead to erroneous conclusions, using a re-analysis of a COVID-19 vaccination study in cancer patients to show that applying a proper target trial emulation framework significantly attenuates the previously reported survival benefits.

The Gist

The "Broken Stopwatch" Problem: Why a Medical Discovery Might Be a Mathematical Illusion

Imagine you are a coach trying to figure out if a new type of high-tech running shoe actually makes athletes faster. You decide to test it by looking at data from a local marathon.

You notice something amazing: the runners wearing the new shoes finished much faster than the runners in regular sneakers. You’re about to write a headline: "New Shoes Boost Speed!"

But then, a mathematician walks up to you and says, "Wait a minute. Your stopwatch is broken."

This paper is that mathematician. It’s not about running shoes or even about COVID-19 vaccines; it’s about how badly timed data can trick scientists into seeing "miracles" that aren't actually there.

---

The Original Story: The "Miracle" Vaccine

A previous study (by a researcher named Grippin) looked at cancer patients receiving a specific type of immunotherapy (a drug that helps the immune system fight cancer). They found that patients who got a COVID-19 vaccine around the same time they started their cancer treatment lived significantly longer.

It sounded like a breakthrough: The vaccine might actually be helping the cancer drug work better!

The Problem: The "Broken Stopwatch" (Time-Related Bias)

The authors of this paper (Dumas and colleagues) looked at the same data and realized the original researchers were using a "broken stopwatch." They identified several ways the timing was messed up:

1. The "Immortal Time" Trap (The VIP Pass)

Imagine you are tracking how long people stay alive. You decide that to be in the "Vaccine Group," a person must receive a vaccine within 100 days of starting treatment.

Here is the trick: To get that vaccine, you have to stay alive for at least those 100 days. If a very sick patient dies on Day 10, they can't get the vaccine, so they are automatically kicked into the "No Vaccine" group.

By definition, your "Vaccine Group" is now filled with people who were healthy enough to survive the first three months. You aren't measuring the vaccine's power; you are accidentally measuring the survivors' strength. It’s like saying, "People who eat breakfast live longer," while ignoring the fact that people who die in their sleep never get the chance to eat breakfast!

2. The "Historical Ghost" (The Time Traveler Problem)

The original study included data from 2015 all the way to 2022. But the COVID vaccine didn't exist until late 2020.

This means the "No Vaccine" group was filled with people from 2015, 2016, and 2017. Medical technology, hospital care, and even the cancer itself change over seven years. Comparing a patient from 2015 to a patient from 2022 is like comparing a person using a flip phone to someone using an iPhone and claiming the iPhone makes you smarter. You aren't comparing the vaccine; you're comparing different eras of medicine.

3. The "Survivor Selection" (The Filter Effect)

If you only look at people who are still around after a certain amount of time, you are looking at a "filtered" group. The original study accidentally filtered out the most fragile patients, making the vaccinated group look much tougher than they actually were.

---

The Fix: The "Target Trial" (The Perfect Simulation)

To find the truth, the authors didn't just look at the old data; they "emulated" a perfect experiment.

Think of it like this: Instead of just looking at old marathon results, they built a digital simulation of a perfectly controlled race. In this simulation:

• Everyone starts at the exact same time.
• No one gets a "head start" just because they survived longer.
• Everyone is from the same era (post-2020).

The Verdict: What is the Truth?

When they ran this "perfect simulation" on the data, the "miracle" vanished.

The original study said the vaccine gave a massive boost to survival. The new, corrected analysis showed that the difference was almost zero.

The takeaway? The vaccine is still great for preventing COVID-19, but it likely doesn't have a "superpower" effect on cancer treatment. The original "miracle" was just a mathematical illusion caused by a poorly timed stopwatch.

---

The Moral of the Story: In science, when you start your clock is just as important as what you are measuring. If your timing is off, your truth will be too.

Technical Summary

Technical Summary: Time-Related Biases in Observational Studies

Problem: Spurious Causal Inference in Observational Data

The paper addresses a critical issue in pharmacoepidemiology: how subtle misalignments in the study timeline—specifically regarding eligibility, intervention assignment, and follow-up—can introduce systematic biases that lead to erroneous conclusions about treatment effects.

The authors use a recent study by Grippin et al. as a case study. Grippin et al. reported that SARS-CoV-2 mRNA (COVID-19) vaccination significantly enhanced the effectiveness of immune checkpoint inhibitors (ICIs) in cancer patients, citing a massive difference in 3-year overall survival (55.7% in vaccinated vs. 30.8% in unvaccinated lung cancer patients). The authors of this paper hypothesize that these findings are likely artifacts of several time-related biases rather than true biological effects.

Methodology: Target Trial Emulation

To investigate this, the authors employ the Target Trial Emulation (TTE) framework. This methodology involves two steps:

1. Specification of a Hypothetical Target Trial: Defining a rigorous, randomized controlled trial (RCT) protocol that would answer the clinical question without bias. This includes precise definitions for the eligibility period, "time zero" (baseline), intervention assignment, and follow-up.
2. Emulation via Observational Data: Using advanced statistical techniques to mimic the RCT using existing longitudinal data.

Key methodological components used in the emulation include:

• Clone-Censor-Weight Approach: To handle the "grace period" (the 100-day window where vaccination can occur), the authors "cloned" patients into both vaccinated and unvaccinated arms at baseline. They then artificially censored the clones if the patient's actual behavior deviated from the assigned strategy.
• Inverse Probability of Censoring (IPC) Weights: Used to adjust for baseline confounding and to account for the selection bias introduced by the censoring process.
• Strict Eligibility Criteria: The authors restricted the cohort to patients initiating ICI as their first systemic treatment and only included those diagnosed after the availability of COVID-19 vaccines (December 2020) to avoid calendar-time confounding.

Key Contributions: Identification of Six Specific Biases

The paper provides a taxonomy of time-related biases that compromised the original study:

1. Immortal-Time Bias: Assigning patients to the vaccine group based on events occurring after time zero (e.g., receiving a vaccine 50 days after ICI start) guarantees they survived long enough to get the vaccine, making the treatment appear protective.
2. Prevalent-User Bias: Including patients who were vaccinated before ICI initiation, which selects for a specific subset of survivors (depletion of susceptibles).
3. Postponed Time Zero Bias: Attempting to fix immortal-time bias by simply delaying the start of follow-up, which still fails to balance susceptible populations.
4. Survivor-Induced Selection Bias: Excluding patients who receive a vaccine later in follow-up, which disproportionately removes those with shorter lifespans from the control group.
5. Differential Time Zero Definition: Setting different baseline dates for different groups (e.g., using the date of the "closest" ICI line for vaccinated patients), which misaligns disease progression stages.
6. Calendar-Time Confounding: Including historical controls from years before vaccines existed, which introduces bias if the standard of care or baseline risk of death changed over time.

Results: Attenuation of the Vaccine Effect

The authors re-analyzed the lung cancer cohort using the emulated target trial protocol. The results showed a dramatic shift in the estimated treatment effect:

• Original Study (Grippin et al.): 3-year OS difference of 24.9 percentage points (55.7% vs 30.8%).
• Emulated Target Trial: 3-year OS difference of only 1.9 percentage points (45.7% vs 43.8%), with a 95% Confidence Interval that crossed zero ([-23.6 to 25.0]).

The re-analysis of the melanoma cohort also showed a substantially smaller effect than originally reported. The results suggest that the apparent benefit of COVID-19 vaccination in the original study was almost entirely driven by the identified time-related biases.

Significance

This paper serves as a vital methodological warning for the medical and epidemiological communities. It demonstrates that even when researchers use "causal inference" methods, failing to align the study timeline with the actual moment of intervention assignment can lead to highly confident but fundamentally incorrect clinical conclusions. The study underscores that Target Trial Emulation is not just a tool for finding effects, but a necessary framework for identifying and preventing "self-inflicted" biases in observational research.