📊 statistics

Testing Partially-Identifiable Causal Queries Using Ternary Tests

This paper proposes using ternary statistical tests to address the limitations of binary tests in hypothesis testing for partially-identifiable causal queries, establishing necessary consistency and topological conditions while demonstrating their application to instrumental variable inequalities and treatment efficacy comparisons.

Original authors: Sourbh Bhadane, Joris M. Mooij, Philip Boeken, Onno Zoeter

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sourbh Bhadane, Joris M. Mooij, Philip Boeken, Onno Zoeter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you only have a blurry, incomplete photo of the crime scene (observational data). You want to know if a specific suspect (a causal model) is guilty.

In the world of statistics, there's a standard tool called a Binary Test. It's like a simple "Guilty" or "Not Guilty" verdict.

Guilty (Reject Null): The evidence is so strong we are sure the suspect did it.
Not Guilty (Don't Reject Null): The evidence isn't strong enough to convict.

The Problem:
Sometimes, the photo is so blurry that two different suspects could have left the exact same footprints. In the paper's language, this is called Partial Identifiability.

If you use a standard Binary Test and say "Not Guilty," you aren't actually proving the suspect is innocent. You're just saying, "The photo is too blurry to tell."
But a standard test forces you to pick one of the two options. It's like a judge saying, "I can't tell if he did it, so I must declare him innocent," which is a dangerous mistake.

The Solution: The Ternary Test (The Three-Option Verdict)
The authors propose a new kind of test that offers three possible outcomes instead of two. Think of it as upgrading your verdict from a simple "Yes/No" to a "Yes / No / I Don't Know" system.

Verdict A (Reject): "The evidence is clear. The suspect is definitely guilty." (The data falls into a zone where only the guilty suspect could have left these footprints).
Verdict B (Don't Reject): "The evidence is clear. The suspect is definitely innocent." (The data falls into a zone where only an innocent suspect could have left these footprints).
Verdict C (Unidentifiable): "The photo is too blurry. The footprints could belong to either the guilty or the innocent suspect. We cannot tell." (The data falls into the overlapping "foggy" zone).

The "Two-Stage" Strategy (The Detective's Toolkit)

You might ask, "How do we build this three-option test? Do we need to invent a whole new science?"

The authors say: No! Use what you already have.

They propose a clever trick called a Two-Stage Ternary Test. Imagine a detective working in two steps:

Step 1: Use a standard "Binary Test" (the old tool) to check if the suspect is in the "Foggy Zone" or the "Clear Zone."
- If the test says "It's the Foggy Zone," you immediately stop and declare "Unidentifiable" (Verdict C).
- If the test says "It's the Clear Zone," you move to Step 2.
Step 2: Use another standard "Binary Test" to decide between "Guilty" and "Innocent" within that Clear Zone.

The paper proves mathematically that this two-step approach is just as powerful as inventing a brand-new, complex three-option machine from scratch. It's like saying, "You don't need a new car; you just need to drive your old car in two specific lanes to get the same result."

Real-World Examples from the Paper

1. The "Broken Instrument" (Instrumental Variable Test)
Imagine you are trying to see if a new medicine works, but you can't run a perfect experiment. Instead, you use a "natural experiment" (like a lottery for who gets the medicine).

The Trap: Sometimes, the lottery is rigged or broken (e.g., only men get the medicine, no women). In this case, the data is "foggy."
The Ternary Solution:
- Stage 1: Check if the lottery was broken (Positivity Test). If yes -> "Unidentifiable."
- Stage 2: If the lottery was fair, check if the medicine actually worked. -> "Guilty" (Works) or "Innocent" (Doesn't work).
- Why it matters: A standard test might falsely claim the medicine works just because the data looked weird. The ternary test stops you from making that mistake by admitting, "Hey, the lottery was broken, so we can't trust the results."

2. The "Threshold Check" (Treatment Efficacy)
Imagine you have a new drug, and you want to know if it's at least as good as a famous competitor drug.

The Trap: The data is noisy. Maybe the new drug is slightly better, slightly worse, or exactly the same.
The Ternary Solution:
- Stage 1: Check if the new drug is clearly worse than the competitor. If yes -> "Innocent" (It's worse).
- Stage 2: If it's not clearly worse, check if it's clearly better. If yes -> "Guilty" (It's better).
- Result: If it's in the middle, the test says "Unidentifiable." This prevents you from making a bold claim when the data is actually just ambiguous.

The "Topological" Secret Sauce

The paper uses some heavy math words like "topology" and "closed sets." In plain English, this is just a way of checking the shape of the possibilities.

Think of the "Guilty" and "Innocent" possibilities as two piles of sand on a beach.

If the piles are separate (no overlap), you can easily tell them apart.
If the piles touch or merge (overlap), you have a "foggy zone."
The authors' math rules tell you: "If the piles are shaped in a specific, solid way (closed sets), you can build a reliable Two-Stage Test. If they are shaped weirdly (like a cloud of dust), you might not be able to build a reliable test at all."

The Big Takeaway

This paper is a guide for scientists and data analysts. It says:

"Stop forcing a 'Yes/No' answer when the data is ambiguous. Instead, use a 'Yes/No/Unknown' system. And the best way to do this? Just combine two simple 'Yes/No' tests in a smart order. This will save you from making false claims when the data is too blurry to be sure."

It turns a frustrating limitation (not knowing for sure) into a useful, honest answer: "We don't know yet." And in science, knowing when you don't know is often the most important discovery of all.

1. Problem Statement

The paper addresses the challenge of hypothesis testing for binary causal queries (e.g., "Is there a direct effect?" or "Is the causal effect zero?") using observational data.

The Core Issue: In many causal settings, queries are partially identifiable. This occurs when the mapping from causal models to observational distributions is not one-to-one. Specifically, the set of observational distributions induced by causal models where the query is true ( $H_1$ ) and the set where it is false ( $H_0$ ) overlap.
Limitation of Binary Tests: Standard binary statistical tests (Reject $H_0$ $H_{0}$ vs. Don't Reject $H_0$ $H_{0}$ ) fail to provide definitive causal conclusions in these overlapping scenarios.
- If a binary test rejects the null, it implies the causal query is true.
- However, if the test fails to reject the null, the result is ambiguous. The data could have come from the intersection of the two sets (where the query is unidentifiable) or from the null set. Thus, a binary test cannot distinguish between "The query is false" and "The query is unidentifiable."

2. Methodology

The authors propose a framework based on ternary (three-outcome) statistical tests to resolve this ambiguity.

A. The Ternary Framework

Instead of a binary output $\{0, 1\}$ , the test outputs one of three results:

Reject Null (0): The data belongs to $R_0 = H_0 \setminus H_1$ (Causal query is definitively 0).
Don't Reject Null (1): The data belongs to $R_1 = H_1 \setminus H_0$ (Causal query is definitively 1).
Unidentifiable (2): The data belongs to $R_2 = H_0 \cap H_1$ (The query cannot be determined from observational data).

B. Error Guarantees and Consistency

The paper defines rigorous error-control desiderata for these tests:

Consistency: As sample size $n \to \infty$ , the probability of correctly identifying the region ( $R_0, R_1, R_2$ ) approaches 1.
Uniform Consistency: The error rates are controlled uniformly across all distributions within the hypotheses.
$(i, j)$ -Error Control: The probability of outputting $i$ when the true distribution is in $R_j$ is bounded by a threshold $\alpha$ .

C. Topological Conditions for Existence

The authors establish necessary and sufficient topological conditions on the hypothesis sets ( $H_0, H_1$ ) for the existence of "good" ternary tests (those satisfying uniform consistency and error control):

Let $W = H_0 \cup H_1$ .
For a specific set of regions $I \subseteq \{R_0, R_1, R_2\}$ to be testable with error control, the sets in $I$ must be closed in the weak topology of probability measures, while the sets not in $I$ must be $F_\sigma$ (countable unions of closed sets).

D. Two-Stage Ternary Tests (Completeness)

To leverage existing binary testing tools, the authors propose constructing ternary tests by combining two binary tests in a two-stage process:

Stage 1: Test a specific region (e.g., $R_2$ ) against its complement.
Stage 2: Depending on the outcome of Stage 1, split the remaining region and perform a second binary test.

Split-Null (SN): First test $R_1$ vs. $R_0 \cup R_2$ .
Split-Alternative (SA): First test $R_2$ vs. $R_0 \cup R_1$ .

Key Theoretical Result: The authors prove that two-stage ternary testing is complete. If a "good" ternary test exists (satisfying the topological conditions), a two-stage construction using binary tests can achieve the same error control. This allows practitioners to use standard binary test libraries to build ternary tests.

3. Key Contributions

Formalization of Ternary Causal Testing: The paper rigorously defines the problem of testing partially identifiable causal queries using a three-outcome framework, explicitly handling the "unidentifiable" case.
Topological Characterization: It provides equivalent topological conditions (closedness vs. $F_\sigma$ ) that determine whether a ternary test with specific error guarantees exists.
Completeness of Two-Stage Construction: It proves that combining binary tests is sufficient to achieve optimal error control for overlapping hypotheses, bridging the gap between theoretical existence and practical implementation.
Practical Guidelines: The authors provide decision trees (based on the topological properties of $R_0, R_1, R_2$ ) to guide practitioners in selecting the correct two-stage test structure (SN vs. SA) for their specific problem.

4. Results and Applications

The framework is applied to two concrete causal inference problems:

A. Testing Instrumental Variable (IV) Inequalities

Context: Testing if an IV model is valid (Pearl, 1995).
Challenge: The intersection $R_2$ corresponds to distributions where the instrument violates the positivity assumption (some instrument values have zero probability).
Solution: The authors construct a Split-Alternative (SA) test.
- Stage 1: Test for positivity (Is $P(Z=z) > 0$ ?). If violated, output "Unidentifiable" (2).
- Stage 2: If positive, test the IV inequalities.
Result: This correctly handles the non-compactness of the positivity region and provides valid error control.

B. Treatment Efficacy Comparison

Context: Comparing a treatment's causal effect ($P(Y=1|do(X=1))$) against a threshold $c$ using observational data.
Challenge: Using Manski bounds, the sets $R_0$ (efficacy $\ge c$ ) and $R_1$ (efficacy $< c$ ) are defined by inequalities.
Solution: The authors construct a Split-Null (SN) test.
- Stage 1: Test if the efficacy is strictly less than $c$ (using a one-sided binomial test on the lower bound).
- Stage 2: If not rejected, test if efficacy is strictly greater than $c$ .
Result: Numerical experiments show that this proposed ternary test achieves higher Probability of Correct Detection (PCD) compared to a naive approach using confidence intervals on Manski bounds.

5. Significance

Bridging Theory and Practice: The paper solves a critical gap in causal inference where standard binary hypothesis testing fails due to partial identifiability. It offers a mathematically sound alternative that provides actionable "Unidentifiable" conclusions rather than ambiguous "Fail to Reject" results.
Leveraging Existing Tools: By proving the completeness of two-stage tests, the authors enable the immediate application of decades of binary hypothesis testing literature to complex causal problems without needing to invent new statistical machinery from scratch.
Robustness: The topological framework ensures that the proposed tests are not just heuristic but are guaranteed to exist and perform correctly under specific, checkable conditions on the data distribution space.

In summary, this work provides a rigorous, topologically grounded methodology for testing causal queries in the presence of unobserved confounding or partial identification, transforming the "unidentifiable" case from a failure of inference into a distinct, testable outcome.