📊 statistics

Assessing covariate-adjusted risk differences in small-sample clinical trials

This paper evaluates methods for estimating covariate-adjusted risk differences in small-sample clinical trials, finding that while standard g-computation approaches often suffer from inflated Type-I error due to misalignment between estimands and variance estimation, robust variants and classical methods like Mantel-Haenszel offer better error control, leading to practical recommendations for aligning inferential targets with appropriate statistical techniques.

Original authors: Martin Schnuerch, Alex Ocampo, Klaus Kähler Holst, Christian Stock

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Schnuerch, Alex Ocampo, Klaus Kähler Holst, Christian Stock

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge trying to decide if a new medicine (Treatment A) works better than a sugar pill (Treatment B). In a perfect world, you'd just give the medicine to half the people and the pill to the other half, count who got better, and compare the numbers. This is the "unadjusted" approach.

But in real life, people aren't identical. Some are older, some have severe disease, and some have mild disease. These differences are like background noise that can make the results look messy. To get a clearer picture, statisticians try to "adjust" for these differences, essentially asking: "If everyone had the same mix of ages and disease severity, would the medicine still win?"

This paper is a massive taste-test competition between ten different statistical "recipes" for doing this adjustment, specifically for small clinical trials (where you only have a few patients, say 30 to 150). The authors wanted to find out which recipe gives the most honest answer without lying about the results.

Here is what they found, explained simply:

1. The Goal: Measuring the "Risk Difference"

Instead of using complex math that is hard to explain (like "odds ratios," which the authors compare to a confusing map that doesn't show the actual terrain), they focused on Risk Difference.

The Analogy: If 20% of people on the sugar pill get better, and 40% of people on the medicine get better, the "Risk Difference" is simply 20%. It's a straight, honest number: "The medicine helps 20 more people out of 100."

2. The Contenders: The Statistical Recipes

The authors tested three main types of methods:

The "Old School" Guards (Mantel-Haenszel & Suissa-Shuster):
These are like experienced, cautious guards. They don't rely on fancy math models that might break.
- Pros: They are incredibly reliable. They almost never cry "Wolf!" when there is no wolf (they rarely make false alarms).
- Cons: They are a bit slow and stubborn. They don't use all the information about the patients' backgrounds, so they sometimes miss a real effect (low power).
The "Modern" Architects (G-Computation):
These are like high-tech architects who build a detailed 3D model of the patients to predict outcomes. They use a computer model (logistic regression) to simulate what would happen if everyone had the same background.
- Pros: They can be very efficient and powerful, especially if you have continuous data (like exact age or blood pressure) rather than just categories (like "young" or "old").
- Cons: In very small groups, their fancy models can get shaky. Sometimes they get so excited they start seeing patterns that aren't there (false alarms).
The "Hybrid" Fixes:
The authors tested several "patches" to fix the shaky Modern Architects. Some used penalties (like a weight on a scale to stop it from wobbling) or robust math (sandwich estimators) to make the models sturdier.

3. The Big Discovery: The "Small Sample" Trap

The most important finding is about small trials (N ≤ 150).

The Problem: When you have very few patients, the "Modern Architect" methods (specifically the standard ones) tend to overestimate their confidence.
- The Metaphor: Imagine a weather forecaster with only three days of data. If they use a standard formula, they might say, "There is a 99% chance of rain!" when it's actually just a 50/50 chance. They are too sure of themselves. In statistics, this is called "Type I error inflation"—saying the drug works when it might not.
The Cause: The paper found this wasn't just because the sample was small; it was because the math used to measure uncertainty didn't match the question being asked. It was like using a ruler to measure weight. The math was "misaligned."

4. The Winners and Losers

The "False Alarm" Kings: Standard G-computation methods (using specific variance formulas) often sounded the alarm too often in tiny trials. They were too eager to find a result.
The "Safe" Bets:
- The Suissa-Shuster test (an exact, non-model-based test) was the most conservative. It rarely made false alarms, but it also missed some real effects because it was so cautious.
- The Mantel-Haenszel test (a classic stratified method) was also very safe and reliable, though it can't handle continuous data well.
The "Balanced" Solutions:
- The authors found that if you take the "Modern Architect" methods and add robust fixes (like the Liu-Xi variance estimator or Firth's penalty), they become much safer. They stop making false alarms, though they become a bit more conservative (less powerful) in exchange for safety.
- Score tests and Bootstrap methods (resampling the data many times) offered a good middle ground, though they still had a slight tendency to be too optimistic in the tiniest samples.

5. The Final Verdict: "Match the Tool to the Job"

The paper concludes that there is no single "best" method for every situation. It depends on what you value most:

If you need absolute safety (no false alarms): Stick to the old-school, design-based tests (Suissa-Shuster or Mantel-Haenszel). They are boring but reliable.
If you want to use patient data to get a sharper answer: You can use the modern G-computation methods, BUT you must use the specific "robust" math formulas (like Liu-Xi or Score tests) that keep the false alarms in check.
The Golden Rule: You must make sure your question (what you are trying to measure), your calculator (the point estimate), and your safety check (the variance) all speak the same language. If they don't match, your results will be misleading, especially in small trials.

In short: In small clinical trials, don't just grab the most complex statistical tool. If you do, you might get a result that looks impressive but is actually a false alarm. You need to pick the tool that is sturdy enough for the small size of your trial and ensure your math isn't lying to you about how sure you are.

Technical Summary: Assessing Covariate-Adjusted Risk Differences in Small-Sample Clinical Trials

Problem Statement
Binary endpoints are ubiquitous in randomized clinical trials (RCTs). Traditionally, conditional odds ratios derived from logistic regression have been the standard for assessing treatment effects. However, odds ratios suffer from interpretability issues, are non-collapsible (meaning marginal and conditional effects diverge even without interactions), and rely on strong modeling assumptions. Recent regulatory guidance, particularly from the US FDA, emphasizes the importance of marginal estimands (population-average treatment effects) and encourages covariate adjustment to improve efficiency.

Consequently, risk differences (RD) have gained prominence as a more interpretable and collapsible measure. However, the methodology for estimating and testing covariate-adjusted risk differences in small-sample settings ( $N \le 150$ ) remains under-explored. Small-sample trials, common in Phase II development, rare diseases, and pediatrics, pose specific challenges: asymptotic approximations often fail, leading to biased estimators, underestimated variances, miscalibrated p-values, and poor confidence interval (CI) coverage. Furthermore, g-computation (standardization) approaches, while flexible, depend on logistic regression working models that may suffer from separation or non-convergence in sparse data. There is a lack of established best practices for aligning the estimand, point estimator, and variance estimator in these contexts.

Methodology
The authors conducted a comprehensive simulation study to evaluate ten statistical methods for estimating and testing risk differences in two-arm RCTs with binary endpoints and prognostic categorical baseline covariates.

Simulation Design: The study mirrored a hypothetical trial with simple randomization ( $1:1$ ), a binary outcome $Y$ , and two binary covariates $X_1, X_2$ . Data were generated via a logistic model with fixed marginal control response probability (0.20) and varying treatment effects ( $\delta = 0, 0.15, 0.30$ ). Covariate prognostic strength and total sample sizes ( $N \in \{30, 60, 90, 120, 150\}$ ) were varied, creating 45 scenarios with 50,000 replications each.
Target Estimand: The simulation framework evaluated performance against the Marginal Treatment Effect (MTE), defined as the average treatment effect over the population distribution.
Methods Evaluated:
1. Unadjusted: Suissa–Shuster exact unconditional test (targets MTE).
2. Stratified (Mantel–Haenszel): Cochran–Mantel–Haenszel (CMH) test (targets Conditional Treatment Effect, CTE); MH Risk Difference with Sato variance (targets Conditional Population-Average Treatment Effect, CPATE); MH Risk Difference with modified Greenland–Robins (mGR) variance (targets MTE).
3. G-Computation (Standardization): Six variants differing in variance estimation and working model adjustments:
  - Ge et al. (model-based delta method; targets CPATE).
  - Liu and Xi (robust sandwich variance + covariate variability term; targets MTE).
  - Ye et al. (semi-parametric robust variance; targets MTE).
  - Steingrimsson et al. (non-parametric bootstrap; targets MTE).
  - Zhang et al. (robust score test; targets MTE).
  - Firth (penalized logistic regression to handle separation; paired with Ge's variance; targets CPATE).

Key Contributions and Results

Computational Stability: Non-convergence of logistic regression models was extremely rare ( $<0.004\%$ ) even at $N=30$ . However, separation (perfect prediction) occurred frequently in small samples with strong prognostic covariates (up to $>30\%$ at $N=30$ ). The study found that separation driven by covariates did not materially degrade the performance of g-computation methods, whereas separation driven by the treatment indicator affected all methods.
Type I Error Control:
- Inflation: Several g-computation approaches using standard Wald-type inference (Ge et al., Ye et al.) exhibited pronounced Type I error inflation in very small samples ( $N=30$ ), primarily due to variance underestimation when the estimand (MTE) and variance estimator (conditional) were misaligned.
- Conservatism: Robust variants (Liu and Xi) and penalized models (Firth) maintained strict Type I error control (often below 0.05) in small samples but at the cost of reduced power.
- Robustness: The Suissa–Shuster test and the CMH test demonstrated excellent Type I error control across all scenarios, though the former was consistently the most conservative.
Statistical Power:
- Methods with inflated Type I error (Ge, Ye) showed the highest power, reflecting their liberal nature.
- Conservative methods (Liu-Xi, Firth, Suissa–Shuster) showed the lowest power in small samples.
- G-computation methods using score tests or bootstrapping offered a middle ground.
- Notably, in the specific setting of categorical covariates, g-computation did not show a substantial power advantage over the MH test when strict Type I error control was required. The authors note that efficiency gains from g-computation are expected to be more pronounced with continuous covariates, which were not included in this simulation.
Estimation and Coverage:
- Bias: Most methods were unbiased. The Firth-based approach introduced a downward bias in the risk difference estimate (due to shrinkage toward 0.50) but achieved lower Root Mean Squared Error (RMSE).
- Coverage: Standard Wald intervals for g-computation (Ge, Ye) and MH risk differences showed substantial undercoverage in small samples. The Liu–Xi and Firth methods provided excellent coverage (near nominal 95%) even in small samples, despite their conservative testing behavior.

Significance and Recommendations
The paper argues that observed Type I error inflation in small-sample covariate-adjusted analyses often reflects a misalignment between the estimand and the variance estimator rather than a fundamental failure of the adjustment method itself. Specifically, using a conditional variance estimator (which ignores covariate distribution variability) to infer a marginal estimand leads to anti-conservative results.

The authors provide the following practical recommendations:

For Strict Type I Error Control: The Suissa–Shuster exact unconditional test is highly robust but loses efficiency by ignoring covariates. For categorical covariates, the CMH test is a robust alternative. If covariate adjustment is necessary, robust g-computation (Liu–Xi variance or Firth model) ensures conservative error control.
For Balancing Power and Error Control: G-computation combined with the Zhang score test or non-parametric bootstrap offers a compromise, showing moderate error inflation but higher power than conservative methods.
For Estimation Accuracy: The Liu–Xi and Firth methods provide the most reliable confidence interval coverage in small samples, though Firth introduces bias.
Alignment is Key: The study emphasizes that method selection must be guided by the target estimand (MTE vs. CPATE) and the inferential framework. G-computation provides a unifying framework for estimation, testing, and interval construction, but the inferential procedure (variance estimator or test statistic) must be explicitly aligned with the chosen estimand to ensure valid inference.

The study concludes that while g-computation is a powerful framework, its application in small samples requires careful selection of variance estimators and working models to avoid inflated error rates, and that no single method dominates across all performance dimensions.