📊 statistics

Real-world performance of large-scale propensity score adjustment strategies: Matching, weighting, and stratification

This study evaluates large-scale propensity score adjustment strategies across four national healthcare databases and concludes that inverse probability of treatment weighting with Crump trimming and 1:1 matching generally offer the best performance, though no single strategy is universally superior, necessitating the use of diagnostics and empirical calibration to guide selection.

Original authors: Kelly M Li, Martijn J Schuemie, Patrick B Ryan, Linying Zhang, Yong Chen, Kashish Priyam, Nicole Pratt, George Hripcsak, Marc A Suchard

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kelly M Li, Martijn J Schuemie, Patrick B Ryan, Linying Zhang, Yong Chen, Kashish Priyam, Nicole Pratt, George Hripcsak, Marc A Suchard

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if a new medicine (let's call it "Drug A") is better than an old one ("Drug B") at treating high blood pressure. In a perfect world, you would run a randomized clinical trial where you flip a coin to decide who gets which drug. This ensures that the two groups of people are identical in every way except for the drug they take.

But in the real world, we can't always flip coins. We have to look at existing medical records. The problem? People who choose Drug A might be very different from those who choose Drug B. Maybe the people on Drug A are older, or sicker, or have different insurance. If you just compare the outcomes directly, you might think the drug is working (or failing) when it's actually just the differences in the patients causing the result. This is called confounding.

To fix this, researchers use a statistical tool called a Propensity Score (PS). Think of the Propensity Score as a "similarity score." It calculates the probability that a specific patient would choose Drug A over Drug B based on hundreds of their characteristics (age, history, other meds, etc.).

The Big Question: How do we use this score?

Once we have these scores, we need to decide how to compare the groups. The paper tests three main ways to do this, like three different ways to organize a messy room:

Matching (The Twin Finder): You take a patient who took Drug A and find a "twin" in the Drug B group who has the exact same similarity score. You pair them up. The paper tested pairing them 1-to-1, 1-to-5, or even 1-to-25.
Stratification (The Sorting Bins): Instead of pairing individuals, you put everyone into 5 (or 25) bins based on their scores. Everyone in "Bin 1" has a low chance of taking Drug A, while everyone in "Bin 5" has a high chance. You then compare the drugs within each bin.
Weighting (The Scale Tilter): You keep everyone, but you give them different "weights" on a scale. If a patient is very rare (e.g., a young person who took Drug A when almost everyone else took Drug B), you give them a heavy weight so their data counts more. If a patient is very common, their weight is lighter. The paper tested different ways to handle the "extreme" weights that can break the scale.

The Experiment: The "SPARTA" Initiative

The authors (from UCLA, Janssen, and others) didn't just guess which method was best. They built a massive testing ground called SPARTA.

The Scale: They looked at 11 million patients across four different national healthcare databases.
The Test: They ran 3,840 different comparisons for 24 different drug matchups.
The "Fake" Outcomes: To know if their methods were actually working, they used 160 "Negative Control" outcomes. These are things the drugs shouldn't affect (like a broken toe or a specific type of allergy). If a method says Drug A causes a broken toe, that method is broken (biased). If it says Drug A has no effect on the broken toe, the method is working.

What They Found

After running thousands of tests, here is the "bottom line" in simple terms:

1. No Single "Magic Bullet"
There isn't one perfect method that wins in every single situation. It depends on the specific data and the specific drugs being compared.

2. The Top Contenders
Two strategies stood out as the most reliable "default" choices:

1-to-1 Matching: Finding one perfect twin for every patient. This was very precise and balanced the groups well.
IPTW with "Crump" Trimming: This is a specific type of weighting where they cut off the most extreme, weird cases (the people who are totally different from everyone else) before weighing the rest. This performed almost as well as matching.

3. The "Don't Do This" List

Stürmer Trimming: This specific way of cutting off extreme cases performed poorly. It removed too many people who actually could be compared, leaving the groups unbalanced.
Stratification (Sorting Bins): While it kept all the data, it often left some "residual bias" (unfairness) inside the bins because people in the same bin weren't actually identical twins.

4. The "Calibration" Secret Sauce
The most surprising finding was about Empirical Calibration.
Imagine you are using a ruler that is slightly bent. You can try to fix the ruler, or you can just measure how bent it is and adjust your final numbers to compensate.
The paper found that if you use a statistical "calibration" step (using the fake outcomes to see how much the method is drifting), all the different methods start to perform almost the same. The differences between them shrink, and the results become much more reliable.

The Takeaway for Researchers

If you are a researcher trying to compare treatments in the real world:

Don't rely on just one method. If you think 1-to-1 matching is best, also try the Crump trimming method as a "sensitivity check" to see if your results hold up.
Check your balance. Make sure the groups you are comparing actually look similar after you apply your method.
Use Calibration. It's like a safety net. It helps fix errors and makes your results less sensitive to which specific method you chose.

In short, while 1-to-1 matching and Crump-trimmed weighting are strong starting points, the best way to ensure your study is trustworthy is to use multiple strategies and "calibrate" your results against known facts.

Technical Summary: Real-world performance of large-scale propensity score adjustment strategies

Problem Statement
Observational studies increasingly utilize large-scale data (millions of patients, tens of thousands of covariates) to estimate treatment effects, often employing Large-Scale Propensity Score (LSPS) models. While LSPS models have been shown to improve covariate balance and model fit through regularized regression, there is a notable lack of empirical evidence regarding how different propensity score (PS) adjustment strategies perform in these large-scale, real-world contexts. Specifically, it remains unclear whether the inclusion of massive numbers of covariates alters the optimal choice among common adjustment strategies: matching, stratification, and inverse probability of treatment weighting (IPTW). Previous benchmarks have yielded conflicting results, with some suggesting IPTW underperforms compared to matching, while others favor IPTW for bias control.

Methodology
The authors conducted the "Survey of Propensity-score Adjustment Routes for Treatment-effect Analyses" (SPARTA), a systematic, large-scale empirical evaluation using real-world data from four federated national healthcare databases (encompassing over 11 million patients).

Study Design: A comparative new-user cohort design was employed, focusing on first-line monotherapies for hypertension. The study evaluated 12 distinct treatment comparisons (drug-level and class-level) against 160 negative control outcomes (where the treatment effect is known to be zero) and 3 synthetic positive controls per negative control.
PS Modeling: LSPS models were fitted using $\ell_1$ -regularized logistic regression with approximately 12,000 covariates per model, covering demographics, medical history, drug exposures, and healthcare utilization. Instrumental variables were screened and removed based on univariate correlation with treatment.
Adjustment Strategies Evaluated:
- Matching: Fixed ratios (1:1, 1:5, 1:25) and variable-ratio (VR) matching with a 0.2 standard deviation caliper.
- Stratification: Grouping patients into 5 and 25 strata.
- IPTW: Three variants were tested:
  1. IPTW (Crump): Trimming the top and bottom 10% of the PS distribution regardless of treatment assignment.
  2. IPTW (Stürmer): A two-step method removing non-overlapping regions and then trimming the most contrary patients.
  3. Overlap Weighting: Up-weighting patients in the region of greatest PS overlap.
Evaluation Metrics: Performance was assessed using Type-I error (false positive rate), Type-II error (false negative rate), mean precision, mean squared error (MSE), and the maximum absolute standardized mean difference (SMD) for covariate balance.
Calibration: An empirical calibration procedure was applied to all estimates to adjust p-values and confidence intervals based on the observed distribution of negative control effects, aiming to capture both random and systematic error.

Key Results

Balance and Bias: Across all model specifications, IPTW with Crump trimming and 1:1 matching consistently achieved the strongest covariate balance (SMD < 0.1) and the lowest Type-I error rates. IPTW (Stürmer) performed poorly across all metrics, exhibiting the highest Type-I error and failing to achieve adequate balance. Stratification showed mixed results, sometimes achieving balance but often at the cost of higher Type-I error.
Precision and MSE: While several methods achieved similar balance, 1:1 matching demonstrated the highest precision and lowest MSE in many scenarios. Interestingly, increasing matching ratios (e.g., 1:25 or variable-ratio) did not consistently improve precision and sometimes introduced variance, likely due to matching less-similar patients and the statistical cost of conditional outcome models.
Impact of Calibration: Empirical calibration significantly reduced the performance differences between strategies. After calibration, Type-I and Type-II error rates, precision, and balance metrics converged across most methods, suggesting that calibration can mitigate the sensitivity of results to the specific choice of adjustment strategy.
Database Variability: Performance varied slightly by database size and composition. For instance, larger matching ratios (1:25) occasionally outperformed 1:1 matching in specific large databases (MDCR, Optum EHR), but 1:1 matching remained a robust default.

Key Contributions

Empirical Benchmarking: The study provides the first large-scale, systematic comparison of PS adjustment strategies (matching, stratification, and multiple IPTW variants) using real-world data with LSPS models across four major healthcare databases.
Differentiation of IPTW Variants: The paper clarifies that not all IPTW methods are equal; it identifies Crump trimming and overlap weighting as highly effective, while demonstrating that Stürmer trimming is suboptimal in large-scale settings due to the removal of the overlapping PS region.
Validation of Calibration: The work empirically validates that empirical calibration effectively standardizes performance across different adjustment strategies, reducing the sensitivity of treatment effect estimates to the specific PS modeling choice.
Practical Guidelines: The authors provide data-driven recommendations for researchers, challenging the notion that a single strategy is universally superior.

Significance and Claims
The paper claims that while no single strategy dominates all settings, IPTW with Crump trimming and 1:1 matching serve as strong default choices for large-scale PS adjustment, offering a favorable balance between bias control and precision. The authors emphasize that researchers should not rely on a single method but should incorporate top-performing strategies as sensitivity analyses. Furthermore, the study strongly advocates for the routine use of empirical calibration to reduce the impact of model choice on study conclusions. The findings suggest that the choice of adjustment strategy should be guided by the specific study context, validated through covariate balance diagnostics, and that the inclusion of many covariates does not preclude the effectiveness of standard adjustment techniques when properly implemented.

The Big Question: How do we use this score?

The Experiment: The "SPARTA" Initiative

What They Found

The Takeaway for Researchers

Technical Summary: Real-world performance of large-scale propensity score adjustment strategies

More like this