Machine learning models for estimating counterfactuals in a single-arm inflammatory bowel disease study

Technical Summary: Machine Learning Models for Estimating Counterfactuals in a Single-Arm IBD Study

1. Problem Statement

In clinical research, Randomized Controlled Trials (RCTs) are the gold standard for estimating treatment effects. However, they are increasingly difficult to conduct due to high costs (often exceeding $100,000 per participant), lengthy recruitment timelines, and ethical concerns regarding assigning patients to placebos or suboptimal standard-of-care (SOC).

Single-arm trials offer a solution by eliminating the concurrent control group, but they lack a comparator to estimate what would have happened to the treated patients under an alternative treatment (the counterfactual). While "external controls" (using historical or contemporaneous cohorts) are a common workaround, they often suffer from selection bias and difficulty in matching distributions. This paper addresses the challenge of estimating treatment effects in a single-arm setting by utilizing virtual controls—machine learning (ML) models trained on external data to predict individualized counterfactual outcomes.

2. Methodology

The researchers used a real-world, multicenter cohort study from the Canadian Children Inflammatory Bowel Disease Network (CIDsCaNN) as a testbed. The study compared two anti-TNF therapies: Infliximab (IFX) and Adalimumab (ADA) in pediatric Crohn’s disease patients.

A. Experimental Design:

• Goal: Use IFX-treated patients to train models that predict what the outcomes of ADA-treated patients would have been if they had received IFX.
• Reference Standard: The researchers compared their ML-derived estimates against a previously published Propensity Score Matching (PSM) analysis of the same cohort.
• Outcomes: 1-year steroid-free clinical remission (SFCR) and a composite of CRP remission plus SFCR (CRP-SFCR).

B. Modeling Framework:
The study evaluated five ML models:

1. Non-pretrained models: Light Gradient Boosting Machine (LGBM), Extreme Gradient Boosting (XGB), Random Forest (RF), and Logistic Regression (LR).
2. Pretrained model: TabPFN (a transformer-based foundation model designed for small tabular datasets).

C. Data Augmentation Strategy:
To combat the relatively small sample size of the IFX cohort, the authors implemented a data augmentation workflow using four generative AI models:

• Adversarial Random Forest (ARF)
• Bayesian Networks (BN)
• Conditional Tabular Generative Adversarial Network (CTGAN)
• Tabular Variational Autoencoder (TVAE)

The models were trained under two settings: Baseline (original IFX data only) and Augmented (original data + synthetically generated records).

3. Key Contributions

• Proof of Concept for Virtual Controls: Demonstrated that a model trained solely on a control arm (IFX) can successfully emulate a single-arm trial for a treatment arm (ADA).
• Evaluation of Data Augmentation: Quantified how synthetic data generation can improve the predictive performance of clinical outcome models.
• Foundation Model Benchmarking: Provided empirical evidence on the utility of pretrained tabular models (TabPFN) versus traditional boosted trees in small-sample clinical contexts.
• Framework for Pretrained Models: Proposed a workflow where a model can be "pretrained" on existing SOC data and "transported" to future single-arm trials.

4. Results

• Performance Gains: Data augmentation significantly improved model performance. For the primary outcome, the relative AUC increased by up to 10% and the Integrated Calibration Index (ICI) improved by up to 39%.
• Best Performing Model: LGBM (specifically when combined with ARF augmentation) yielded the best balance of discrimination (AUC) and calibration (ICI). It also produced Odds Ratios (OR) closest to the PSM reference standard.
• TabPFN Observations: While TabPFN showed the highest baseline performance on small data, it derived minimal additional benefit from synthetic augmentation, likely due to its extensive prior training on millions of synthetic datasets.
• Treatment Effect Estimation: All models (both baseline and augmented) produced 95% Confidence Intervals that included 1.0, aligning with the original PSM study's conclusion: there was no statistically significant difference in effectiveness between ADA and IFX.

5. Significance

This study validates the use of virtual controls as a viable, cost-effective, and potentially more efficient alternative to traditional external controls in Inflammatory Bowel Disease (IBD) trials. By showing that ML models can accurately estimate counterfactuals, the research paves the way for "portable" virtual control models. Such models could be deployed in future single-arm trials to reduce the need for expensive and ethically complex concurrent control recruitment, provided they undergo rigorous external validation and assessment of transportability.