📊 statistics

Finite-sample bias-variance tradeoff with variables related to trial participation inserted into causal forest models for ensuring generalizability

This paper demonstrates that while including trial-participation covariates in causal forest models theoretically enables unbiased conditional average treatment effect estimation, in finite RCT samples the resulting variance inflation often degrades performance, making inverse probability weighting a more effective strategy for generalizing results to broader populations.

Original authors: Rikuta Hamaya, Etsuji Suzuki, Konan Hara

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rikuta Hamaya, Etsuji Suzuki, Konan Hara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Club" vs. The "Crowd"

Imagine you are trying to figure out if a new medicine works. You run a clinical trial, which is like a special club. Only certain people get to join this club (the trial participants). Maybe the club is only open to people who live nearby, have good insurance, or are very healthy.

Now, you want to know: "Will this medicine work for the entire population (the 'crowd'), not just the people in the club?"

The problem is that the people in the club are different from the people in the crowd. If you just look at the club members, your results might be biased (skewed) because the club isn't a perfect mirror of the real world.

The Goal: Finding the "Personalized" Effect

Doctors don't just want to know the average effect of a drug. They want to know the Conditional Average Treatment Effect (CATE). In plain English, this means: "How does the drug work specifically for a 60-year-old smoker with high blood pressure?"

To do this, researchers use a powerful computer tool called a Causal Forest. Think of a Causal Forest as a team of thousands of tiny detectives (decision trees) that look at many different clues (covariates) to predict who will benefit from the medicine.

The Dilemma: The "Too Many Clues" Trap

The researchers in this paper asked a specific question: To make our predictions accurate for the whole crowd, should we feed the Causal Forest every single clue we have, including the reasons why people joined the club in the first place?

The Theory: Yes, you should. If you tell the forest, "People who joined the club were mostly rich and lived in cities," the forest can mathematically adjust for that and give you a fair answer for the whole crowd.
The Reality (The Paper's Finding): In real-world scenarios with limited data (typical medical trial sizes), adding all those extra clues actually makes the prediction worse.

The Analogy: The Overloaded Chef

Imagine you are a chef (the Causal Forest) trying to cook the perfect soup (the treatment effect) for a huge banquet (the general population).

The Simple Approach (Model 1): You only use the main ingredients you know affect the taste (like salt and pepper). You cook a lot of batches. The soup tastes okay, but maybe a little off because you didn't account for the fact that your kitchen is in a humid city (selection bias).
The "Perfect" Theory (Model 2): You decide to be perfect. You add every possible ingredient to the pot: salt, pepper, humidity levels, the color of the walls, the chef's mood, and the brand of the spoon.
- The Problem: Because you are adding so many random ingredients, the soup becomes chaotic. The flavor fluctuates wildly from batch to batch. Sometimes it's too salty, sometimes it's bland. The variance (the inconsistency) is so high that the soup is actually less reliable than the simple version.
The Paper's Solution (IPW): Instead of throwing everything into the soup pot, you use a strainer (Inverse Probability Weighting). You take the soup you made with just the main ingredients and "strain" it to remove the bias caused by the humid kitchen.
- Result: You get a consistent, reliable soup that tastes right for the whole banquet, without the chaos of adding too many random ingredients.

What the Paper Actually Found

The researchers ran computer simulations to test this. Here is what they discovered:

The Trade-off: When they added the "club membership" clues (variables related to trial participation) directly into the Causal Forest, the math became very unstable. The variance (the noise) grew so large that it canceled out the benefit of fixing the bias.
The Sample Size Issue: This only worked well if the trial was massive (thousands of people). For typical medical trials (hundreds or a few thousand people), adding those extra variables made the predictions less precise.
The Winner: The best strategy was to use a method called Inverse Probability Weighting (IPW). This method handles the "club vs. crowd" difference before the Causal Forest does its work. It allowed the forest to focus on the important clues without getting overwhelmed by the noise.

The Real-World Test

To prove this wasn't just a computer game, the researchers applied their method to a real study about Omega-3 fish oil and heart disease.

They compared the results of the "add everything" method vs. the "IPW" method.
They found that the IPW method shifted the results to better reflect what would happen in the general population, while the "add everything" method created too much noise to be useful.

The Bottom Line

If you are trying to predict how a drug works for the general public using data from a specific clinical trial:

Don't just dump every single piece of data you have into your machine learning model.

While it sounds logical to include the reasons people joined the trial to fix the bias, doing so often creates too much "noise" for the model to handle, especially with normal-sized studies. Instead, it is smarter to fix the bias separately (using a weighting method like IPW) and then let the model focus on the most important factors.

Key Takeaway: In the world of medical data, sometimes less is more. Adding too many variables to fix a problem can actually make the prediction worse.

Technical Summary: Finite-Sample Bias-Variance Tradeoff in Causal Forests for Generalizable CATE Estimation

Problem Statement
Estimating Conditional Average Treatment Effects (CATE) from Randomized Controlled Trials (RCTs) to generalize findings to broader source populations is critical for personalized medicine. However, this process is complicated by selection bias arising from trial participation and the challenges of high-dimensional covariates. While identification theory suggests that unbiased CATE estimation in the source population is achievable by conditioning on covariates related to trial participation (denoted as $X_2$ ) alongside effect modifiers of interest ( $X_1$ ), the practical application of this theory using machine learning methods like Causal Forests remains unclear. Specifically, there is a concern that in finite samples typical of medical RCTs, the variance inflation caused by including high-dimensional selection covariates may outweigh the benefits of bias reduction.

Methodology
The authors conducted a systematic evaluation of finite-sample bias-variance tradeoffs using simulation studies and a real-world application.

Data Generation: The simulation assumed an RCT setting where participants ( $S=1$ ) are selected from a larger source population ( $S=0$ ) based on covariates $X_2$ . The true Individual Treatment Effect (ITE) was defined as a function of effect modifiers ( $X_{1,ALL}$ ), selection covariates ( $X_2$ ), and unobserved modifiers ( $O$ ). Trial sample sizes ranged from 200 to 5,000, with source populations of 100,000.
Estimation Approaches: The study compared four distinct strategies for estimating CATE using Causal Forests:
1. Model 1 (Naïve): Causal Forest fitted on trial data using only $X_1$ .
2. Model 2 (Direct Inclusion): Causal Forest fitted on trial data using both $X_1$ and $X_2$ . For estimating $CATE(X_1)$ , Monte Carlo integration was used to marginalize over $X_2$ .
3. Model 1IPW: Causal Forest fitted on trial data weighted by Inverse Probability Weighting (IPW) derived from a logistic regression of $X_2$ , using only $X_1$ in the forest.
4. Model 2IPW: Causal Forest fitted on IPW-weighted trial data using both $X_1$ and $X_2$ .
Evaluation Metrics: Performance was assessed via Mean Squared Error (MSE), bias, and variance for two aims:
- Aim A: Estimating $CATE(X_1)$ in the source population.
- Aim B: Predicting ITE via $CATE(X_1, X_2)$ in the source population.
Real-World Application: The methods were applied to the VITAL trial (omega-3 fatty acids and coronary heart disease) using NHANES data as the source population to illustrate the practical impact of these strategies.

Key Results

Bias-Variance Tradeoff: Theoretical identification suggests that including $X_2$ (Model 2) yields unbiased estimates. However, simulations demonstrated that in realistic RCT sample sizes, the variance inflation from adding $X_2$ to the Causal Forest often outweighed the modest reduction in bias.
Sample Size Sensitivity: Including $>3$ covariates related to participation ( $X_2$ ) substantially degraded precision (increased MSE) unless sample sizes were very large. For $X_1$ with $\ge3$ covariates, Model 2 required significantly larger sample sizes to outperform Model 1.
Superiority of IPW: IPW-based methods (Model 1IPW and Model 2IPW) consistently improved performance across scenarios. Model 1IPW, in particular, offered a favorable bias-variance compromise, often outperforming the naïve approach (Model 1) even when selection bias was present.
Application Findings: In the VITAL trial application, IPW-adjusted models shifted CATE estimates toward source population effects and refined heterogeneity assessments compared to unweighted models.

Key Contributions and Significance
The paper challenges the conventional assumption that simply including trial-selection variables within CATE-estimating algorithms (like Causal Forests) is the optimal strategy for generalizability. The authors highlight a critical finite-sample limitation:

Variance Inflation: Directly inserting trial-participation variables into Causal Forests can inflate estimator variance to a degree that reduces overall prediction performance (MSE), particularly in medical RCTs with sample sizes under 5,000.
Strategic Recommendation: The authors propose that addressing selection bias separately—specifically through Inverse Probability Weighting (IPW)—is a more reasonable strategy than direct inclusion. This approach allows the Causal Forest to focus on effect modifiers ( $X_1$ ) while IPW corrects for the selection mechanism, thereby maintaining lower variance while achieving necessary bias correction.
Theoretical Grounding: The findings are framed within the bias-variance tradeoff of non-parametric, tree-based estimators, noting that expanded dimensionality degrades convergence rates and dilutes the random subspace feature-sampling process, forcing suboptimal tree construction.

Limitations
The authors acknowledge that their conclusions are based on simulations within a specific context and that the empirical application could not be fully validated against source population outcomes due to the lack of counterfactual data in real-world settings. Furthermore, they note that their comparison did not include pseudo-outcome approaches, as those are not yet readily extendable to high-dimensional CATE estimation in their current form.