📊 statistics

Scalable Counterfactual Risk Estimation for Rare Events in Longitudinal Data

This paper proposes a principled subsampling and reweighting strategy to address the computational burden and estimation instability caused by rare outcomes in longitudinal causal inference, demonstrating its effectiveness through simulations and a large-scale EHR study on suicide risk.

Original authors: Xiaohui Yin, Avijit Mitra, Ying Zhou, Kun Chen, Hong Yu

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaohui Yin, Avijit Mitra, Ying Zhou, Kun Chen, Hong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Did a specific treatment (like a new medication or a lifestyle change) actually cause a patient to survive longer, or did they survive just by luck?

In the world of medical research, this is called causal inference. Usually, researchers look at huge databases of patient records (longitudinal data) to find the answer. They track patients over time, watching how their health changes, what treatments they get, and whether bad things happen.

The Problem: The "Needle in a Haystack" Nightmare

The paper focuses on a specific, tricky scenario: Rare Events.

Imagine you are looking for a specific type of rare disease or a suicide event in a database of 130,000 people. Most people in the database are healthy (the "haystack"), and only a tiny handful experience the event (the "needle").

To figure out if a treatment works, researchers use a complex mathematical recipe called the g-formula. Think of this recipe as a giant, multi-step simulation. To get an accurate answer, they have to:

Build a statistical model for every single time point in the study.
Run this model on the entire database of 130,000 people.
Repeat this whole process hundreds of times (using a technique called "bootstrapping") to make sure the answer isn't just a fluke.

The Catch: Because the event is so rare, the computer spends 99% of its time processing the 129,000 healthy people who didn't have the event. It's like trying to find a needle in a haystack by examining every single piece of hay one by one, even though you know the needle is only in a tiny corner. This takes forever and crashes computers, especially when you need to run the simulation hundreds of times to be sure.

The Solution: The "Smart Sampling" Strategy

The authors propose a clever shortcut: Longitudinal Case-Control Subsampling with Reweighting.

Here is the analogy:
Instead of examining the entire haystack, you decide to:

Keep every single needle (every person who had the event).
Pick a small, random handful of hay (a sample of the healthy people) to represent the rest of the haystack.
Do the math on this smaller pile.

But there's a catch: If you just throw away the extra hay, your math will be wrong because you've changed the proportions. You can't just count the needles and the few pieces of hay you picked; you have to pretend the handful of hay represents the whole mountain.

The "Magic Scale" (Reweighting):
The paper introduces a special weighting system. Think of it like a magic scale.

If you pick 1 healthy person to represent 1,000 healthy people in the real world, you tell the computer: "This one person counts as 1,000."
The authors figured out the exact mathematical "weights" needed to balance the scales at every single time step of the study. They ensure that even though they are looking at a small sample, the final result is mathematically identical to what they would have gotten if they had looked at the whole database.

Why This is a Big Deal

Speed: By looking at a tiny fraction of the healthy people (the "hay") but keeping all the sick people (the "needles"), the computer finishes the job 4 to 10 times faster. In their real-world test with veterans, a calculation that took 24 seconds dropped to just 5 seconds.
Accuracy: Despite using less data, the results were almost exactly the same as the full database. The "magic scale" (weights) fixed the math so the answer didn't get distorted.
Stability: When events are rare, trying to fit a model on a massive dataset of mostly healthy people can make the math "wobble" and fail to converge. By balancing the numbers with their sampling method, the math becomes much more stable and reliable.

The Real-World Test

The authors tested this on a massive study of 127,399 US veterans to see if social and behavioral factors (like housing or employment) influenced suicide risk.

The Event: Suicide is tragically rare (only 331 cases in the study).
The Result: Using their "Smart Sampling" method, they got the same risk estimates as the full study but in a fraction of the time. This means researchers can now run these complex, life-saving analyses on massive datasets without waiting weeks for the computer to finish.

Summary

The paper says: "Don't waste time counting every single healthy person when looking for rare events. Keep all the sick people, pick a smart sample of the healthy ones, and use a special mathematical 'scale' to make the small sample act like the big one. You get the same answer, but much, much faster."

Technical Summary: Scalable Counterfactual Risk Estimation for Rare Events in Longitudinal Data

Problem Statement

Estimating the causal effect of time-varying treatments on survival outcomes in large observational studies is computationally prohibitive when outcomes are rare. Standard approaches based on the g-formula, such as the Iterative Conditional Expectation (ICE) and Non-Iterative Conditional Expectation (NICE) estimators, require fitting regression models at every time point across the entire cohort. In settings with rare events (e.g., suicide risk), this leads to severe class imbalance, causing instability and slow convergence in logistic regression. Furthermore, valid inference typically requires bootstrap-based variance estimation, which involves repeated resampling and model fitting, exponentially increasing the computational burden. In secure or memory-constrained environments, processing full longitudinal cohorts repeatedly is often infeasible.

Methodology

The authors propose a principled longitudinal case–control subsampling and reweighting strategy designed to reduce computational costs while preserving statistical consistency. The method is applicable to a broad class of longitudinal causal estimators, with a specific focus on the ICE estimator.

Core Mechanism:

Outcome-Dependent Subsampling: Instead of analyzing the full cohort, the method retains all individuals who experience the event (cases) at each time point $t$ . Controls (individuals who remain event-free) are subsampled at a specific ratio $J_t$ (e.g., 5, 10, or 20 controls per case).
Longitudinal Constraints: Unlike standard case–control designs that operate locally, this approach respects the temporal structure of longitudinal data. Sampling decisions at one time point impose constraints on subsequent points due to survival and within-subject dependence. The method constructs a coherent sampled dataset that accounts for these cross-time dependencies.
Reweighting for Consistency: To ensure the subsample yields consistent estimates of the full-population counterfactual risk, the authors derive specific weights ( $w_t, m_t, \ell_t$ $w_{t}, m_{t}, ℓ_{t}$ ) for cases, sampled controls, and censored individuals.
- The weights are derived by equating the expected total weighted contribution of individuals across mutually exclusive groups (e.g., those dying at time $t$ , those censored at time $t$ , and those surviving).
- The authors demonstrate that under these balance conditions, the weighted estimating equations of the subsample are unbiased estimators of the population-level estimating equations.
Algorithm Integration: The proposed weighting scheme is integrated into the standard ICE algorithm (Algorithm 2). At each backward iteration step (from $T$ to 0), the current global time point is treated as a local baseline, and the appropriate weights are applied to the regression fitting of the conditional expectation models.

Handling Censoring:
The framework explicitly handles administrative censoring and loss to follow-up. Censored individuals are included in the subsample at time points prior to their censoring, with weights derived to balance their contribution against cases and controls.

Key Contributions

Scalable Framework: The paper introduces a general subsampling framework that can be applied to existing g-formula-based estimators (ICE and NICE) to drastically reduce computational load in rare-event settings.
Theoretical Consistency: The authors provide a theoretical proof (Theorem 1) showing that the proposed subsampling estimator remains consistent for the target counterfactual risk $E(Y^g_T)$ under standard causal assumptions (consistency, exchangeability, positivity), provided the derived weighting balance equations are satisfied.
Practical Weighting Scheme: The paper offers a concrete, flexible weighting scheme that allows for varying sampling ratios across time points, accommodating scenarios where event rates change over the study duration.

Results

The authors evaluated the method through extensive simulations and a real-world application.

Simulation Studies:

Setup: Simulated longitudinal data with $N=30,000$ individuals over 6 time points, with a low event prevalence (~1%).
Computational Efficiency: The subsampling approach achieved a dramatic reduction in computation time. For logistic regression, the total runtime was reduced by approximately 60–70% (e.g., from ~11.8s to ~5.8s for NICE). For more complex machine learning models (SuperLearner), the reduction was even more pronounced, dropping from ~63 minutes to ~3–5 minutes.
Statistical Performance: The risk estimates from the subsampled data were comparable to the full-data counterparts. Increasing the control-to-case ratio ( $J$ ) from 5 to 20 reduced standard errors, bringing them closer to the full-sample estimates. The boxplots of bootstrap means showed that the subsampling versions tracked the complete-data results closely.
Robustness: The method remained effective in medium-prevalence settings (~2.5% and ~6%), though the computational advantage naturally diminished as the event rate increased.

Real-World Application:

Dataset: A cohort of 127,399 U.S. veterans discharged from psychiatric hospitals, followed for suicide risk (331 events) over five 6-month intervals.
Performance: The subsampling ICE estimator ran in roughly one-quarter of the time required for the complete-data version (5.73s vs. 24.66s per fit).
Accuracy: Point estimates for suicide risk under different treatment strategies (always treated vs. never treated) were similar between the two methods. The subsampling approach showed slightly larger standard errors at the final time point, consistent with the reduced sample size, but maintained overall validity.

Significance and Claims

The paper claims that the proposed method offers a scalable alternative to full-cohort estimation for longitudinal causal inference in rare-event settings. Its primary significance lies in enabling the application of rigorous g-formula-based causal inference to massive datasets where full-sample estimation is computationally infeasible, particularly when bootstrap resampling is required for inference.

The authors emphasize that:

The method preserves consistency without requiring new identification assumptions beyond those of the standard g-formula.
It alleviates the computational bottleneck caused by repeated model fitting on highly imbalanced risk sets.
It is flexible, allowing for adaptive sampling ratios and various weighting schemes as long as the balance equations are met.
The approach is particularly valuable for online or routine analyses where models must be refitted frequently as new data arrives, as the cumulative time savings from bootstrap resampling are substantial (e.g., reducing a 6.9-hour bootstrap process to 1.6 hours).

The paper concludes that while the subsampling step reduces cost, it does not alter the target estimand, making it a practical tool for researchers dealing with large-scale, rare-outcome longitudinal data.