📊 statistics

Residual-on-Residual Regression as a Tool for Effect Estimation in Observational Data

This paper proposes residual-on-residual regression as a stable, interpretable, and computationally simple alternative to AIPW and TMLE for estimating exposure effects in observational data, demonstrating through simulations and real-world application that it performs comparably to established methods under standard conditions and outperforms them when positivity violations occur.

Original authors: Ashley I. Naimi, Qianhui Jin, Ya-Hui Yu, Sara M. Parisi, Lisa M. Bodnar

Published 2026-07-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ashley I. Naimi, Qianhui Jin, Ya-Hui Yu, Sara M. Parisi, Lisa M. Bodnar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if eating a lot of vegetables actually helps prevent a specific pregnancy complication called preeclampsia. You have a huge pile of data from thousands of pregnant women. But there's a catch: the women who eat lots of vegetables are also different in many other ways—they might be wealthier, exercise more, or have better access to healthcare. These differences are called "confounders." If you just compare the two groups directly, you might think the vegetables are doing the work, when really it's just the other healthy habits.

To solve this, scientists use statistical "magic tricks" to strip away those other differences and isolate the effect of the vegetables. This paper introduces a specific magic trick called Residual-on-Residual Regression and shows that it works just as well as the two most popular tricks currently used (called AIPW and TMLE), but with a few extra perks.

Here is how the paper breaks it down, using simple analogies:

The Problem: The "Noise" in the Signal

Think of the data as a radio broadcast. You want to hear the "vegetable signal" (the effect of the diet), but the radio is full of static (the confounders like age, income, and exercise).

Old Way: Scientists used to try to write a perfect script to describe exactly how every piece of static sounds. If they guessed the script wrong, the result was wrong.
New Way (Machine Learning): Now, scientists use "Super Learners" (computer algorithms) to learn the pattern of the static automatically. But these learners are so flexible they can sometimes get confused or unstable if the data is tricky.

The Three Contenders

The paper compares three methods for cleaning up the radio signal:

AIPW & TMLE: These are the current "champions." They are very powerful and can fix mistakes if one part of their calculation is wrong (a feature called "doubly robust"). However, they can get shaky if the data has gaps (for example, if almost no one in the study eats vegetables, making it hard to compare).
Residual-on-Residual Regression: This is the "underdog" method that the authors are championing. It's like a clever two-step cleaning process.

How "Residual-on-Residual" Works

Imagine you want to know if a specific type of shoe makes you run faster. But people who wear those shoes also tend to be taller, and being taller helps you run faster.

Step 1 (The Clean-Up): First, you predict how fast a person should run based only on their height (ignoring the shoes). You then calculate the difference between their actual speed and that prediction. This difference is called a "residual." It's the "leftover" speed that height couldn't explain.
Step 2 (The Repeat): You do the exact same thing for the shoes. You predict how many people should wear those shoes based on their height. You calculate the "leftover" (residual) shoe-wearing that height couldn't explain.
Step 3 (The Match-Up): Now, you take the "leftover speed" and the "leftover shoe-wearing" and see if they are connected. Since you already removed the "height" factor from both sides, any connection you find is purely about the shoes.

The paper calls this "Residual-on-Residual" because you are regressing (comparing) the leftovers against the leftovers.

What the Study Found

The authors tested this method in two ways:

1. The Real-World Test (The nuMoM2b Study)
They looked at real data from nearly 8,000 pregnant women to see if high vegetable intake reduced preeclampsia.

The Result: All three methods (the two champions and the underdog) agreed. They all found that high vegetable intake was linked to a small but real reduction in preeclampsia risk. The numbers were so close that they confirmed each other.

2. The Simulation Test (The Stress Test)
They created fake data on a computer to see how the methods handle trouble.

The "Messy Data" Test: When the data was complex and non-linear (like a tangled knot), all three smart methods did a great job, while a simple, old-fashioned method failed miserably.
The "Missing Data" Test: This was the big reveal. They created a scenario where the "positivity" assumption was violated—meaning there were huge gaps in the data (e.g., almost no one in a certain group ate vegetables).
- The AIPW and TMLE methods started to wobble and became less precise.
- The Residual-on-Residual method stayed steady and calm. It didn't get confused by the gaps.

Why This Matters

The paper argues that Residual-on-Residual Regression is a fantastic tool for researchers because:

It's Stable: It doesn't freak out when data is missing or uneven, unlike its competitors.
It's Simple: It's easier to code and understand than the complex "champion" methods.
It's a Safety Net: If you use all three methods and they agree, you can be very confident in your result. If they disagree, it's a red flag that your model might be broken.

The Catch (The Fine Print)

The paper notes one important rule: This method assumes the effect of the vegetables is the same for everyone (a "constant effect"). If the vegetables helped some women but hurt others, this specific method might give a slightly different answer than the others. However, for most standard questions where we want to know the "average" effect, it works beautifully.

In summary: The paper says that while the fancy, complex methods (AIPW and TMLE) are great, the simpler "Residual-on-Residual" method is a reliable, stable, and easy-to-use alternative that often performs just as well, and sometimes even better when the data is messy. It's a great tool to have in your statistical toolbox.

Technical Summary: Residual-on-Residual Regression as a Tool for Effect Estimation in Observational Data

Problem Statement
Researchers frequently utilize observational data to estimate causal exposure effects, yet standard parametric regression models often fail in high-dimensional settings where the number of potential confounders is large relative to the sample size. In such contexts, correctly specifying functional forms and interactions is difficult, and misspecification can introduce biases that standard sensitivity analyses may not detect. While machine learning (ML) algorithms offer data-adaptive alternatives for modeling complex confounder relationships, they typically optimize a loss function with a bias-variance tradeoff. If unaccounted for, the bias introduced by flexible ML methods can degrade the performance of naïve singly robust causal estimators.

Although semiparametric efficiency theory has produced doubly robust estimators like Augmented Inverse Probability Weighting (AIPW) and Targeted Maximum Likelihood Estimation (TMLE) that can recover optimal statistical properties, these methods are not without limitations. Specifically, they can be highly sensitive to even minor violations of the positivity assumption (e.g., when sample sizes are insufficient to populate all confounder strata with both exposed and unexposed individuals). Furthermore, applying both TMLE and AIPW to the same dataset can yield divergent results, creating uncertainty regarding which estimate to trust.

Methodology
The paper proposes and evaluates Residual-on-Residual Regression, a method rooted in Robinson's (1988) work and popularized in econometrics, as a stable alternative for estimating average treatment effects (ATE) within a partially linear model framework.

The approach operates under the specification:
$Y = \psi A + g(W) + \epsilon$
where $Y$ is the outcome, $A$ is the exposure, $W$ is a set of confounders, $\psi$ is the constant treatment effect, and $g(W)$ is an unspecified nonparametric function. Unlike standard regression, this model assumes a linear relationship between exposure and outcome but leaves the confounder-outcome relationship flexible.

The estimation proceeds in three steps:

Nuisance Estimation: Fit nonparametric models (e.g., using Super Learner ensembles) to estimate the conditional mean of the outcome given confounders, $\hat{E}[Y|W]$ , and the conditional mean of the exposure given confounders, $\hat{E}[A|W]$ .
Residual Construction: Compute residuals for both the outcome ( $\tilde{Y} = Y - \hat{E}[Y|W]$ ) and the exposure ( $\tilde{A} = A - \hat{E}[A|W]$ ).
Final Estimation: Estimate $\psi$ by regressing $\tilde{Y}$ on $\tilde{A}$ using a no-intercept Ordinary Least Squares (OLS) estimator.

To ensure valid inference and robustness, the authors employ out-of-sample validation (cross-fitting) to prevent overfitting in the nuisance models. Variance estimation utilizes the HC3 robust (sandwich) variance estimator, which corresponds to the empirical variance of the estimator's Neyman-orthogonal influence function.

Empirical Application and Simulation Results
The authors applied this method to data from the Nulliparous Pregnancy Outcomes Study: Monitoring Mothers-to-Be (nuMoM2b) ( $n=7,923$ ) to estimate the association between high vegetable intake density and preeclampsia.

Empirical Findings: Residual-on-residual regression, AIPW, and TMLE yielded concordant estimates, all indicating a modest reduction in preeclampsia risk associated with high vegetable intake. The residual-on-residual approach produced an adjusted risk difference of -1.49 per 100 participants (95% CL: -2.94, -0.03), comparable to AIPW (-0.97) and TMLE (-1.15).

A Monte Carlo simulation study ( $n=500$ , 10,000 replicates) compared the finite-sample properties of residual-on-residual regression against AIPW, TMLE, and a misspecified linear-additive OLS model.

Primary Simulations (Positivity Met): When the true effect was constant and confounder relationships were nonlinear, all three doubly robust estimators were essentially unbiased with near-nominal confidence interval coverage. They substantially outperformed the misspecified parametric OLS model, which exhibited severe bias and collapsed coverage.
Positivity Stress Tests: In scenarios with induced positivity violations (where propensity scores approached 0 or 1), residual-on-residual regression demonstrated superior stability. While AIPW's root mean squared error (RMSE) quadrupled and TMLE's coverage dropped to 88.7% at severe violation levels, residual-on-residual regression remained essentially invariant in bias, precision, and coverage. This stability is attributed to the method's reliance on residualizing the exposure rather than inverting propensity scores, which avoids the instability associated with extreme weights.

Key Contributions and Significance
The paper argues that residual-on-residual regression offers a practical, interpretable, and computationally simple tool for causal inference that complements AIPW and TMLE. Its significance lies in several areas:

Stability under Positivity Violations: Unlike AIPW and TMLE, which rely on inverse probability weighting, residual-on-residual regression does not require inverting the propensity score. This makes it more robust when the positivity assumption is weak or violated, a common issue in fields like nutritional epidemiology.
Triangulation Strategy: Because the method shares doubly robust properties with AIPW and TMLE but utilizes a different estimation mechanism, it serves as a valuable triangulation tool. Disagreement between these estimators can signal model dependence or violations of underlying assumptions, warranting further investigation.
Interpretability and Implementation: The method yields a closed-form coefficient and robust standard error, making it easy to implement in standard software. It provides a direct estimate of a constant treatment effect under the partially linear model assumption.

The authors note that the interpretability of the estimator as an ATE hinges on the assumption of a constant treatment effect. If effect heterogeneity exists, the estimator converges to a variance-weighted average effect rather than the simple ATE. However, the authors suggest that in settings where the constant effect assumption is plausible, residual-on-residual regression provides a stable, efficient, and robust alternative to existing doubly robust methods.