Simulation of synthetic health records for assessment of causal inference methods for vaccine efficacy

Technical Summary: Simulation of Synthetic Health Records for Assessment of Causal Inference Methods for Vaccine Efficacy

Problem Statement
Public health agencies increasingly rely on observational data from electronic health records (EHRs), such as the EAVE-II platform in Scotland, to evaluate vaccine effectiveness and safety in real-world settings. However, traditional observational methods often fail to draw valid conclusions regarding counterfactual scenarios (outcomes that would have occurred under alternative interventions) due to confounding variables like age, sex, and comorbidities. While formal causal inference methods (e.g., Marginal Structural Models) can address these issues, their development and testing are severely hindered by privacy and governance constraints. Researchers typically cannot access individual-level patient data for methodological development until specific permissions are granted, creating a bottleneck that delays the creation of robust analysis pipelines and the training of new researchers.

Methodology
The authors developed a framework to generate low-fidelity synthetic health records that explicitly encode known causal and confounding mechanisms, parameterized using aggregated data from the EAVE-II platform.

• Data Source and Parameterization: The simulation was grounded in aggregated EAVE-II data (extracted August 2025) covering nearly the entire Scottish population (N ≈ 4.57 million). Demographic distributions (sex and age) were used to simulate confounders ($Z$), while vaccination counts informed treatment allocation models.
• Causal Framework: The study utilized the simulation approach proposed by Seaman and Keogh, which extends the work of Evans and Didelez. This framework defines the joint probability distribution through four components:
  1. The distribution of confounders $P(Z=z)$.
  2. The causal effect of treatment on the outcome $P^*(Y=y|A=a)$.
  3. A scalar "risk score" function $h(Z, \bar{A}_k)$ ranking individuals by confounder-driven risk.
  4. An association parameter $\rho_k$ determining the strength of dependence between the risk score and the outcome.
• Simulation Design:
  • Confounders: Sex (binary) and Age (16 ordinal categories) were encoded as a vector $Z$.
  • Treatment: A single clinic visit representing vaccine allocation ($A_0 \in \{0,1\}$) was simulated using logistic models derived from EAVE-II data.
  • Outcomes: Five rare adverse events were modeled: thrombocytopenia, immune thrombocytopenic purpura (ITP), venous thromboembolism, arterial thromboembolism, and haemorrhagic stroke. "Ground truth" parameters for these outcomes were derived from clinical events extracted from EAVE-II (November 2025).
  • Confounding Scenarios: Five distinct scenarios were generated by varying the association parameter $\rho_k$ from $-0.1$ (weak confounding) to $-0.9$ (strong confounding).
  • Scale: For each of the 25 scenario-outcome combinations (5 scenarios $\times$ 5 outcomes), 1,000 synthetic datasets of 100,000 individuals were generated.
• Analysis: The authors tested the recovery of true causal parameters using two approaches: unweighted logistic regression and Inverse Probability of Treatment Weighting (IPTW) logistic regression.

Key Results

• Weak Confounding: In scenarios with weak confounding ($\rho_k = -0.1$), both unweighted and IPTW-weighted logistic regression models successfully recovered the true causal parameters.
• Strong Confounding: As the strength of confounding increased ($\rho_k$ approaching $-0.9$), unweighted models exhibited increasing bias and coverage rates far below the nominal 95%.
• Robustness of Weighted Models: In contrast, the IPTW-weighted models consistently recovered the true causal parameters across the full range of confounding strengths, demonstrating only small residual bias as $\rho_k$ approached its lower limit.
• Workflow Efficiency: The study demonstrated that using synthetic data allows analysts to implement and test full analysis pipelines in parallel with the data access application process, rather than waiting for approval.

Significance and Claims
The paper claims that these synthetic datasets serve as a critical resource for the EAVE-II team and other public health researchers to:

1. Develop and Test Pipelines: Build and validate causal inference pipelines (specifically Marginal Structural Models) without requiring access to sensitive individual-level data.
2. Train Researchers: Provide a realistic training environment for analysts to learn causal inference techniques for vaccine effectiveness research.
3. Accelerate Research: Reduce delays caused by data governance processes by allowing methodological development to proceed before real data access is granted.

The authors explicitly state that these datasets are intended only for pipeline development, testing, and training. They are not designed to answer clinical or public health questions, as they are parameterized from aggregated statistics and do not reproduce the individual-level structure of the underlying records. Consequently, the authors caution against using these synthetic data to draw clinical or policy conclusions.

Limitations
The authors note a specific limitation regarding the simulation framework: the Seaman and Keogh method assumes treatment allocation can be assessed at each visit with probabilities strictly between 0 and 1. Due to the nature of the EAVE-II data (weekly outcomes following a single dose, where re-vaccination is impossible within a one-week interval), the authors were restricted to simulating a single baseline visit for treatment allocation and one subsequent visit for outcome assessment. They identify the need for future work to perform more complex longitudinal simulations involving multiple sequential visits.