📊 statistics

Application of Propensity Score Models and Causal Estimators in Observational Studies under Model Misspecification

This study demonstrates through extensive simulations and real-world applications that Augmented Inverse Probability Weighting (AIPW) offers the most robust and stable causal effect estimates in observational studies under model misspecification, particularly when integrated with flexible machine learning approaches for propensity score estimation.

Original authors: Apu Chandra Das, Sakib Salam, Md Robiul Islam Talukder, Ashim Chandra Das, Antar Chandra Das, Rakhi Chowdhury

Published 2026-05-21

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Apu Chandra Das, Sakib Salam, Md Robiul Islam Talukder, Ashim Chandra Das, Antar Chandra Das, Rakhi Chowdhury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a new fertilizer makes plants grow taller. In a perfect world, you would flip a coin for every plant: heads, it gets the fertilizer; tails, it doesn't. This is a Randomized Controlled Trial (RCT). Because the coin flip is random, the plants that get the fertilizer are, on average, identical to the ones that don't, except for the fertilizer itself.

But in the real world (observational studies), we can't always flip coins. Maybe the plants that choose to get the fertilizer are already the ones with richer soil or more sunlight. If you just compare the heights, you might think the fertilizer worked, when really it was just the better soil. This is called confounding.

To fix this, statisticians use a tool called a Propensity Score (PS). Think of the Propensity Score as a "matchmaker" or a "similarity score." It calculates how likely a plant was to get the fertilizer based on its soil, sunlight, and water. If a plant with poor soil got the fertilizer, the score helps us realize it was an "outlier" and gives it extra weight in our analysis to balance the scales.

The Problem: Guessing the Matchmaker's Rules

The tricky part is that we don't know the true rules the matchmaker used. We have to build a model to guess them.

The Old Way: Use a simple, rigid rulebook (like Logistic Regression). It's easy to read, but if the real world is messy and complex, the rulebook might be wrong (Model Misspecification).
The New Way: Use a super-smart, flexible AI (like Random Forests or SVM). It can learn complex patterns, but sometimes it gets too excited and makes wild guesses, leading to unstable results.

The paper asks: Which method works best when our guess about the rules is wrong?

The Three Main Tools Tested

The researchers tested three different ways to use these scores to find the "true" effect of the treatment:

RSM (The "Outcome" Detective): This method ignores the matchmaker entirely. It just looks at the plants that got fertilizer and tries to predict their height based on their soil.
- Weakness: If your model of how soil affects height is wrong, your answer is wrong.
IPW (The "Weight" Balancer): This method uses the matchmaker's score to create a "fake population." It gives heavy weights to plants that are rare (e.g., a plant with poor soil that got fertilizer) and light weights to common ones.
- Weakness: If the matchmaker's score is slightly off, the weights can become crazy (like giving one plant a weight of 1,000,000), making the whole calculation wobble and break.
AIPW (The "Double-Check" Safety Net): This is the hybrid. It uses both the matchmaker (to balance the weights) AND the outcome detective (to predict the height).
- Superpower: It has a "Double Robust" property. This means you only need one of the two guesses to be right. If the matchmaker is wrong but the outcome detective is right, it still works. If the outcome detective is wrong but the matchmaker is right, it still works.

What the Simulations Showed

The researchers ran thousands of computer simulations where they knew the "true" answer but pretended they didn't. They messed up the rules in different ways to see which tool survived.

When everything was perfect: All tools worked well, but the "Double-Check" (AIPW) was very stable.
When the Matchmaker (PS) was wrong:
- The Weight Balancer (IPW) crashed. It became very unstable, especially when using the fancy AI methods, because the AI made wild guesses that created huge, unstable weights.
- The Outcome Detective (RSM) kept working fine because it didn't rely on the matchmaker.
- The Double-Check (AIPW) kept working fine because it had the Outcome Detective to back it up.
When the Outcome Detective was wrong:
- The Outcome Detective (RSM) failed completely.
- The Weight Balancer (IPW) worked only if the matchmaker was simple and correct. If the matchmaker was a fancy AI, IPW failed again.
- The Double-Check (AIPW) kept working because it had the matchmaker to back it up.

The Big Takeaway: The AIPW (Double-Check) method was the most reliable. It was the only one that didn't care which part of the model was broken, as long as one part was right. The fancy AI methods (like Random Forests) were great at finding patterns, but they were dangerous to use alone with the Weight Balancer (IPW) because they could create unstable results.

Real-World Tests

The authors tested these tools on two real datasets:

ACTG175 (The "Fair" Test): This was a real medical trial where patients were randomly assigned to treatment. Since the assignment was random, there was no bias to fix.
- Result: All the tools agreed with each other. This proved that the tools work correctly when the conditions are fair.
ADNI (The "Unfair" Test): This was a study on Alzheimer's disease where people weren't randomly assigned; they were already sick or healthy. This is a messy, real-world scenario with lots of confounding.
- Result: The tools disagreed!
  - The Weight Balancer (IPW) said the exposure (having Alzheimer's) had a huge negative effect on cognition.
  - The Double-Check (AIPW) said the effect was much smaller and less certain.
  - The Outcome Detective (RSM) said there was almost no effect.
- Why? Because the data was messy, the simple Weight Balancer got confused by extreme weights. The Double-Check method used its "safety net" to smooth things out, giving a more conservative and stable answer.

The Bottom Line

If you are trying to figure out cause-and-effect in messy real-world data:

Don't just rely on one method.
Fancy AI models are powerful but can be unstable if used alone for weighting.
The Double-Check (AIPW) method is the safest bet. It combines the best of both worlds, ensuring that even if your model for "who gets treated" is wrong, or your model for "what happens next" is wrong, you still have a good chance of getting the right answer.

It's like having a backup parachute: if your main one fails, the second one saves you. In statistics, that backup is the "Double Robust" property.

Technical Summary: Application of Propensity Score Models and Causal Estimators in Observational Studies under Model Misspecification

Problem Statement
Propensity score (PS) methods are standard tools in observational studies for reducing confounding and estimating causal treatment effects. However, the validity of PS-based estimators is heavily contingent upon the correct specification of the underlying models. In practice, the true functional forms of both the treatment assignment (PS) and outcome models are typically unknown. Misspecification—arising from incorrect functional forms, omitted variables, or unaccounted interactions—can lead to substantial bias, instability, and increased variability in causal estimates. While machine learning (ML) methods (e.g., Random Forests, Support Vector Machines) offer flexible alternatives to traditional parametric models (e.g., logistic regression) for PS estimation, their impact on the performance of classical causal estimators under varying degrees of model misspecification remains insufficiently understood. Specifically, there is a need to evaluate how flexible ML-based PS estimation interacts with Response Surface Modeling (RSM), Inverse Probability Weighting (IPW), and Augmented Inverse Probability Weighting (AIPW) when models are misspecified.

Methodology
The study employs a comprehensive simulation framework and two real-world case studies to evaluate estimator performance.

Simulation Design:
- Data Generation: The authors generated independent observations with a binary treatment indicator ( $A$ ), a continuous outcome ( $Y$ ), and a vector of baseline covariates ( $X$ ) of dimension $p=9$ . Covariates followed a multivariate normal distribution with varying correlation structures ( $\rho \in \{0.2, 0.7\}$ ).
- Misspecification Scenarios: Four distinct regimes were constructed by combining correct and incorrect specifications for both the PS and outcome models:
  1. Both models correctly specified.
  2. PS misspecified (omitted confounders and added interaction terms), outcome correctly specified.
  3. PS correctly specified, outcome misspecified (omitted covariates).
  4. Both models misspecified.
- Estimators and PS Models: The study compared three causal estimators:
  - RSM: Outcome regression only.
  - IPW: Weighting based on the PS.
  - AIPW: Doubly robust estimator combining outcome regression and weighting.
  - PS Estimation Methods: Logistic Regression (LR), Random Forests (RF), Linear Discriminant Analysis (LDA), and Support Vector Machines (SVM).
- Evaluation Metrics: Performance was assessed across 1,000 Monte Carlo replicates using bias, absolute bias, Root Mean Squared Error (RMSE), empirical standard error (SE), and 95% confidence interval width. Sample sizes of $n=200$ and $n=1000$ were tested.
Real-World Applications:
- ACTG175: A randomized clinical trial (HIV-1) used as a benchmark where all estimators should theoretically converge, serving as a calibration check.
- ADNI: A retrospective observational study (Alzheimer's Disease) reorganized as a quasi-experiment to estimate the effect of disease status (LMCI/AD vs. Cognitively Normal) on cognitive decline (ADAS13 score change).

Key Contributions and Results

Superiority of Doubly Robust Estimators:
Across all simulation scenarios, the Augmented Inverse Probability Weighting (AIPW) estimator consistently demonstrated robust and stable performance. Due to its "doubly robust" property, AIPW provided consistent estimates as long as either the PS model or the outcome model was correctly specified. Even when both models were misspecified, AIPW exhibited lower bias and more stable variance compared to single-model estimators.
Vulnerability of IPW to Flexible PS Estimation:
The IPW estimator showed high sensitivity to PS misspecification and the choice of PS estimation method. When flexible machine learning methods (particularly Random Forests and SVM) were used to estimate the PS, IPW produced substantial bias and instability. This was attributed to ML methods yielding extreme propensity scores near 0 or 1, resulting in unstable weights. IPW only performed well when the PS model was correctly specified using parametric methods (LR, LDA).
Limitations of RSM:
RSM performed well only when the outcome model was correctly specified. Under outcome model misspecification, RSM exhibited substantial bias, confirming that outcome-only approaches lack robustness to violations of the outcome model assumptions.
Impact of Sample Size and Correlation:
Increasing the sample size from 200 to 1000 significantly improved estimator performance (reduced bias, RMSE, and SE) across all methods, consistent with asymptotic theory. Higher covariate correlation ( $\rho=0.7$ ) generally led to improved estimation accuracy, likely due to enhanced predictive strength of the included covariates.
Real-World Findings:
- In the ACTG175 (randomized) trial, all estimators (IPW, AIPW, RSM) yielded consistent, positive, and statistically significant treatment effects, validating the methods under ideal identification conditions.
- In the ADNI (observational) study, significant discrepancies emerged. IPW estimators consistently indicated a negative exposure effect (greater cognitive decline in the exposed group), while AIPW and RSM estimates were attenuated, with confidence intervals including zero. This highlighted the variance-bias trade-off: IPW was sensitive to residual confounding and extreme weights, whereas AIPW provided more conservative estimates by leveraging outcome modeling to correct for PS misspecification.

Significance and Claims
The paper claims that the integration of flexible machine learning approaches within doubly robust frameworks (specifically AIPW) is critical for improving causal effect estimation in observational studies. The authors argue that while ML methods can capture complex treatment assignment mechanisms, their direct application in weighting-based estimators (IPW) can be problematic due to instability. Conversely, when paired with AIPW, the flexibility of ML methods can be exploited without compromising estimator robustness.

The study concludes that the choice of PS estimation method has a substantial impact on causal inference, particularly for weighting-based estimators. It advocates for the use of doubly robust estimators like AIPW as a reliable alternative that maintains stable performance across a wide range of model specifications. These findings are presented as relevant for clinical trials and regulatory science, where methodological transparency and robustness are essential for credible decision-making in the era of real-world data. The authors modestly note limitations, including the focus on moderate-dimensional data and the assumption of fully observed confounders, acknowledging that performance may differ in ultra-high-dimensional settings or under unmeasured confounding.