📈 economics

Testing the identification of causal effects in observational data

This paper proposes a machine learning-based test for a conditional independence condition that, if satisfied, simultaneously validates an instrumental variable and confirms the unconfoundedness of treatment effects in observational data, demonstrating its application to the study of fertility's impact on female labor supply where the test suggests the instrument is invalid.

Original authors: Martin Huber, Jannis Kueck

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Huber, Jannis Kueck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if a specific action (let's call it the Treatment) actually causes a specific result (the Outcome). For example, does taking a new vitamin (Treatment) actually make you run faster (Outcome)?

In a perfect world, you would run a controlled experiment: you'd flip a coin to decide who gets the vitamin and who doesn't. But in the real world, we often only have observational data—we just watch what people do on their own. The problem is that people who choose to take vitamins might already be healthier or more disciplined than those who don't. This makes it hard to tell if the vitamin caused the speed or if the person's natural discipline did.

Usually, statisticians have to guess that they have accounted for all the other factors (like diet, sleep, and age) that might be messing up the results. They say, "We assume that once we control for these things, the treatment is as good as random." But until now, there was no way to actually test if that guess was right.

The Detective's New Tool: The "Suspected Instrument"

This paper introduces a clever new way to test that guess. The authors suggest using a third variable, which they call a "Suspected Instrument."

Think of the Suspected Instrument like a weather vane that points to the Treatment but shouldn't directly affect the Outcome.

The Setup: Imagine you want to know if eating breakfast (Treatment) improves test scores (Outcome).
The Suspected Instrument: You suspect that the time the school bus arrives (Instrument) determines whether a kid eats breakfast (they might skip it if the bus is late).
The Logic: The bus arrival time shouldn't directly make a kid smarter or dumber. It only affects the test score through whether they ate breakfast.

The Big Test: The "Conditional Independence" Check

The authors discovered a mathematical rule that acts like a litmus test. They say:

"If we look at people who ate breakfast (Treatment) and we control for their age and income (Covariates), the Bus Arrival Time (Suspected Instrument) should have zero connection to their Test Scores (Outcome)."

If the bus arrival time is still connected to the test scores even after we account for breakfast and other factors, then something is wrong. It means either:

The bus time actually affects test scores in some other way (maybe late buses make kids stressed?), OR
The "breakfast" group wasn't actually random to begin with (maybe the bus schedule is linked to the neighborhood's wealth, which affects test scores).

The Magic: If this test passes (the bus time is truly unrelated to the scores once we control for everything), then two things are proven true at once:

Your "Suspected Instrument" is a valid tool.
Your "Treatment" (breakfast) is effectively random, meaning you can trust your results about the causal effect.

If the test fails, you know your causal claim is shaky.

How They Do It: The "Double Machine Learning" Robot

In the past, doing this test required simple math that couldn't handle complex data with hundreds of variables (like age, income, education, neighborhood, weather, etc.).

The authors used Double Machine Learning (DML). Imagine a super-smart robot that:

Learns the patterns: It looks at all the complex data to figure out how the Bus Time, Breakfast, and Test Scores are all related to each other.
Removes the noise: It strips away the influence of all the other factors (the "covariates") to see the pure relationship between the Instrument and the Outcome.
Checks the result: It runs a statistical test to see if any connection remains.

They call it "Double" because the robot learns two things simultaneously (how the instrument works and how the outcome works) and uses a special technique called "cross-fitting" to make sure the robot doesn't just memorize the data (overfitting) but actually learns the rules.

The Real-World Test: Kids, Siblings, and Jobs

To prove their method works, the authors applied it to a famous study about fertility and women's jobs.

Treatment: Having a third child.
Outcome: How much the mother works.
Suspected Instrument: The "sibling sex ratio" of the first two kids. The idea is: if parents want a mix of boys and girls, having two kids of the same sex might make them try for a third child.

The authors ran their new test on this data. They found that the "sibling sex ratio" was not independent of the mother's work hours, even after controlling for things like the father's income and the mother's age.

What this means: The test failed. This suggests that either the "sibling sex ratio" isn't a perfect random tool (maybe families with two boys have different financial situations than families with two girls), OR that having a third child isn't as random as we thought once we control for those factors. In short, the old study's conclusion might be flawed because the "randomness" assumption didn't hold up under their new microscope.

Summary

This paper gives researchers a new, powerful magnifying glass. Instead of blindly trusting that they have controlled for all the "confusing factors" in their data, they can now use a suspected "instrument" to run a test. If the test passes, they can be confident in their cause-and-effect conclusions. If it fails, they know they need to rethink their assumptions. They built this tool using advanced machine learning to handle complex, real-world data.

Technical Summary: A Testable Condition for Treatment Effect Identification

Problem Statement
In observational studies, causal inference typically relies on identifying assumptions that are conventionally deemed untestable. The most prominent of these is the "selection-on-observables" (or unconfoundedness/ignorability) assumption, which posits that the treatment is as good as randomly assigned conditional on observed covariates. Whether a specific set of covariates is sufficient to satisfy this assumption is usually justified by theory or intuition rather than empirical evidence. This paper addresses the gap by demonstrating the existence of a testable condition for the identification of causal effects in observational data, provided that the data includes both a set of covariates to be controlled for and a "suspected instrument."

Methodology and Theoretical Framework
The authors propose a framework that jointly tests the validity of a suspected instrument and the selection-on-observables assumption for the treatment. The approach relies on a specific causal structure (Assumption 1) characterized by:

No Reverse Causality: The outcome does not cause the treatment, covariates, or the suspected instrument; the treatment does not cause the covariates or the instrument.
Causal Faithfulness: Statistical independence implies d-separation (no causal paths), and vice versa.
Common Support: All combinations of treatment and instrument values exist conditional on covariates.
Conditional Dependence: The suspected instrument is statistically associated with the treatment conditional on covariates (first-stage effect).

Under these conditions, the paper establishes a key theoretical result (Theorem 1): The conditional independence of the suspected instrument ( $Z$ ) and the outcome ( $Y$ ) given the treatment ( $D$ ) and covariates ( $X$ ) is a necessary and sufficient condition for the joint satisfaction of:

Instrument Validity (A): $Z$ does not directly affect $Y$ and is unconfounded conditional on $X$ .
Selection-on-Observables (B): $D$ is unconfounded conditional on $X$ .

Consequently, if the testable condition $Y \perp\!\!\perp Z | D, X$ holds, both assumptions are satisfied, and the treatment effect is identified. The paper also extends this to conditional mean independence (Theorem 2) for assessing Average Treatment Effects (ATE).

Testing Approaches
To test the conditional independence condition in high-dimensional settings, the authors propose methods based on Double Machine Learning (DML) (Chernozhukov et al., 2018), which allows for data-driven control of covariates while maintaining desirable asymptotic properties.

Doubly Robust (DR) Estimation: The authors construct a test based on the average difference in conditional means ( $\Delta$ ). They utilize a doubly robust score function that combines estimates of the conditional mean outcome and the instrument propensity score. By employing cross-fitting, they achieve root- $n$ consistency and asymptotic normality even when using flexible machine learning algorithms for nuisance parameter estimation.
Heterogeneity Detection: To address the issue that averaging might mask violations, the authors propose an algorithm to detect heterogeneity in violations across covariate values. This involves splitting the sample, using machine learning (e.g., decision trees) to identify subgroups where violations are likely, and testing conditional independence within these subgroups.
Global Squared Difference Test: A second approach tests the global null hypothesis that the squared difference in conditional means is zero ( $E[(\mu(1, D, X) - \mu(0, D, X))^2] = 0$ ). This method uses an $L_2$ -type aggregate measure and introduces an auxiliary random variable to ensure the test statistic has a non-degenerate distribution under the null.

Simulation Results
The paper evaluates the finite sample performance of these tests via simulations with 50 covariates and sample sizes of 1,000 and 4,000.

Empirical Size: Under the null hypothesis (where assumptions hold), the tests generally maintain empirical rejection rates close to the nominal 10% level.
Power: The tests demonstrate decent power in detecting violations. The DR-based test is effective when violations are uniform, while the squared difference test shows high power even when violations are heterogeneous across covariates.
Multiple Testing: When testing for heterogeneity across subgroups, the authors find that while power increases with sample size, multiple testing corrections (Bonferroni or multiplier bootstrap) are necessary to control false discovery rates, though they may reduce power in smaller subsamples.

Empirical Application
The methodology is applied to US Census data (Angrist and Evans, 1998) to evaluate the impact of fertility on female labor supply, using the sex ratio of the first two children as a suspected instrument.

Findings: The tests reject the null hypothesis of conditional independence. Specifically, the squared difference test yields a p-value close to zero, and the DR test is significant at the 5% level.
Interpretation: The results suggest a violation of the testable implication. Given the moderate set of socio-economic covariates (mother's age, father's income, etc.), this implies that either the selection-on-observables assumption fails, the sibling sex ratio is not a valid instrument, or both. The heterogeneity analysis indicates that violations are particularly pronounced among families with higher father's income.

Significance and Claims
The paper claims to provide a practical solution for a long-standing problem in causal inference: the inability to test the core assumptions required for identification in observational data.

Joint Testing: Unlike previous literature that often assumes a valid instrument to test selection-on-observables (or vice versa), this approach tests both conditions jointly. It does not require the researcher to a priori believe the instrument is valid; rather, the instrument serves as an auxiliary variable to test the validity of the selection-on-observables assumption.
Flexibility: By leveraging Double Machine Learning, the approach accommodates high-dimensional covariates and non-linear relationships without relying on parametric assumptions that might be misspecified.
Caveats: The authors are modest regarding the scope of their claims. They note that the testable condition relies on specific causal assumptions (faithfulness, no reverse causality). Furthermore, while the test can detect violations, it does not pinpoint the exact source of failure (instrument invalidity vs. unobserved confounding of treatment) without further assumptions. The approach is framed as a diagnostic tool to assess the credibility of identification strategies rather than a method to generate new causal estimates in the presence of violations.