A simple strategy for valid inference in target trial emulations

1. Problem Statement

Target trial emulation is a structured framework for causal inference using observational data, where researchers specify a hypothetical randomized trial (the "target trial") and attempt to replicate its protocol using real-world data. While this approach clarifies causal questions and reduces biases (e.g., immortal time bias), it faces a critical statistical challenge: the iterative nature of protocol specification.

In practice, investigators rarely define the target trial protocol a priori. Instead, they iteratively refine the protocol (eligibility criteria, treatment strategies, outcomes, and statistical models) after inspecting the observational data to ensure feasibility and identifiability. This creates two main threats to statistical validity:

1. Data Snooping/Selective Inference: When the same dataset is used to both select the protocol (e.g., choosing a specific treatment comparison or outcome based on preliminary effect sizes) and estimate the effect, conventional confidence intervals lose their nominal coverage guarantees. The inference becomes biased because the model was "tuned" to the specific noise in that dataset.
2. Invalid Coverage Rates: Standard statistical theory assumes the analysis plan is fixed before data inspection. Iterative refinement based on data features (e.g., dropping covariates due to convergence issues, excluding subgroups with rare treatments, or selecting outcomes with larger observed effects) inflates Type I error rates and invalidates p-values and confidence intervals.

2. Methodology: Sample Splitting

The paper proposes a sample splitting procedure to decouple protocol selection from effect estimation, ensuring valid statistical inference even when the protocol is data-driven.

The Procedure:

1. Random Split: Before any analysis, the dataset is randomly divided into two independent samples:
   • The Specification Sample: Used for exploratory analysis.
   • The Emulation Sample: Used for the final estimation and inference.
2. Phase 1 (Specification): Investigators use the Specification Sample to:
   • Assess data availability and variable definitions.
   • Determine identifiability (e.g., checking for unmeasured confounding).
   • Evaluate feasibility (e.g., checking sample sizes for rare treatments).
   • Select the final target trial protocol (eligibility, treatment strategies, outcomes) and the specific statistical estimator/model.
   • Crucially, all decisions regarding the causal estimand and analysis plan are finalized here.
3. Phase 2 (Emulation): The finalized protocol is applied to the Emulation Sample (which was never seen during the selection phase).
   • The effect estimate and confidence intervals are computed strictly on this second sample.
   • Because the protocol is fixed relative to the Emulation Sample, standard statistical inference (e.g., 95% confidence intervals) retains its nominal coverage guarantees.

Analogy: The authors liken this to the clinical drug development pipeline, where pilot/early-phase studies (Specification Sample) inform the design of a Phase 3 trial (Emulation Sample), which is then analyzed independently.

3. Key Contributions

• Resolution of Iterative Bias: The paper provides a simple, practical solution to the problem of "data-dependent protocol selection." It demonstrates that valid inference is possible even when investigators use data to refine eligibility, treatment definitions, and model specifications, provided the final estimation occurs on a held-out sample.
• Distinction from Machine Learning Splits: The authors clarify that this approach differs from standard cross-fitting or training/validation/test splits in machine learning. In target trial emulation, the first split is used to choose the estimand and the protocol structure itself, not just to fit model hyperparameters. Therefore, standard cross-fitting (which assumes the model structure is fixed) is not applicable.
• Addressing Selective Reporting: The method mitigates the risk of "estimand picking" (selecting the protocol that yields the most promising results). Even if investigators explore multiple protocols in the Specification Sample and choose the one with the largest effect, the final inference on the Emulation Sample remains unbiased regarding that specific choice.
• Clarification of Limitations: The paper explicitly notes that while sample splitting fixes statistical validity (coverage), it does not fix clinical validity. If the protocol is revised to a population that is easier to analyze but less clinically relevant, the result is still valid for that specific population, but may not answer the original clinical question.

4. Results and Theoretical Justification

The paper argues theoretically that sample splitting restores the validity of confidence intervals in four specific scenarios where conventional full-sample inference fails:

1. Feasibility Constraints: When protocols are revised based on variable availability (e.g., removing an outcome not recorded in the database).
2. Identifiability Constraints: When protocols are revised because key confounders are missing (e.g., switching to a treatment comparison where confounding is better controlled).
3. Precision/Feasibility Constraints: When subgroups are excluded due to rare treatments or when model convergence issues force a switch to parametric assumptions.
4. Selective Reporting: When investigators explore multiple protocol variations and report only the most "interesting" ones.

In all these cases, because the selection process is confined to the Specification Sample, the Estimation Sample acts as a "fresh" dataset where the protocol is effectively pre-specified. Thus, the coverage probability of the confidence intervals remains at the nominal level (e.g., 95%).

5. Significance and Limitations

Significance:

• Bridging Practice and Theory: It aligns the reality of observational research (which is inherently iterative) with the rigorous requirements of statistical inference.
• Transparency: It encourages a transparent workflow where investigators must document the exploratory findings and choices made in the Specification Sample, similar to reporting pilot study results.
• Accessibility: Unlike complex adaptive inference methods (e.g., differential privacy or selective inference algorithms), sample splitting is conceptually simple and easy to implement using standard statistical software.

Limitations:

• Loss of Power: The primary drawback is that only a fraction of the data (e.g., 80-90%) is used for the final estimation, reducing statistical power. The authors suggest a small Specification Sample (10-20%) is often sufficient for protocol design.
• No Cross-Fitting: Standard cross-fitting cannot be used because the "model" being selected is the entire causal protocol, not just a nuisance parameter.
• Prior Knowledge Bias: The method does not fully protect against bias if investigators have prior knowledge of the data from previous analyses (e.g., earlier emulations of similar trials), as the "unseen" data is not truly unseen to the researcher's intuition. However, the authors argue this bias is likely smaller than the bias from using the same data twice for selection and estimation.

Conclusion:
The paper concludes that sample splitting is a robust, simple strategy to ensure that target trial emulations yield valid statistical inferences, even when the protocol is refined based on data inspection. It mirrors the gold standard of randomized trial development (pilot to Phase 3) and should be adopted to prevent the inflation of false-positive rates in comparative effectiveness research.