📊 statistics

On the Impossibility of Specification Testing of Interference Models Based on Exposure Mappings

This paper establishes that no specification test for exposure mapping interference models can simultaneously control Type I and Type II errors when distinguishing between nested models, proving that any such test performs no better than random guessing unless the alternative hypothesis is further restricted, as demonstrated by a consistent test for differentiating no-interference from network-linear-in-means models.

Original authors: Chao Gao, Christopher Harshaw, Fredrik Sävje, Yitan Wang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chao Gao, Christopher Harshaw, Fredrik Sävje, Yitan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery in a crowded room. You suspect that people are influencing each other's behavior (this is called "interference" in statistics). To solve the case, you need a theory about how they are influencing each other. Maybe you think, "They only talk to their immediate neighbors," or "They only copy the person sitting directly across from them."

In the world of statistics, these theories are called Exposure Models. They are simplified maps that try to explain the complex web of interactions in an experiment.

This paper, written by four researchers, delivers a surprising and somewhat disappointing verdict: You cannot use the experiment's data alone to prove that your map is the correct one.

Here is the breakdown of their findings using simple analogies.

1. The "Magic Map" Problem

Imagine you are trying to figure out if a specific map of the room is accurate.

The Null Hypothesis (Your Map): You believe the map says, "People only talk to their left neighbor."
The Alternative (The Real World): The reality might be, "People talk to their left neighbor and the person three seats away."

The researchers ask: Can we look at the data from the experiment and say, "Aha! Your map is wrong!"?

2. The "Naive Detective" Result

The paper's main finding is a "strong impossibility result." It says that for any map based on these "exposure" rules, there is no test you can build that is better than a naive detective who flips a coin.

Here is the analogy:
Imagine you have a test that claims to detect if your map is wrong. The researchers prove that for every possible way the real world could differ from your map, there is a "trick" scenario where the data looks exactly the same whether your map is right or wrong.

Because the data looks identical in these trick scenarios, your test cannot tell the difference.

If you try to make your test sensitive enough to catch the "trick" scenarios, you start making false alarms (Type I errors) when your map is actually correct.
If you try to stop the false alarms, you become blind to the "trick" scenarios (Type II errors).

The paper proves that the sum of your mistakes (false alarms + missed clues) will always equal 100%. This means your test is no better than a detective who closes their eyes, flips a coin, and says, "I'm rejecting your map!" 50% of the time.

The Takeaway: You cannot use the experiment's data to validate the entire structure of how people influence each other if you are relying on these standard "exposure maps." The data simply doesn't contain enough information to distinguish between a correct map and a slightly wrong one in the worst-case scenario.

3. Why Does This Happen? (The "Chameleon" Effect)

The researchers explain that these exposure models are like chameleons. You can change the underlying rules of the game (the "true" way people influence each other) in a way that perfectly mimics the rules of your map, making them indistinguishable to the data.

It's like trying to tell if a magician is using a specific trick or a different one, but the magician is so skilled that the audience sees the exact same result either way. No matter how many times you watch the show (how large your sample size is), you can't figure out which trick was used just by looking at the final outcome.

4. Is All Hope Lost? (The "Special Case" Exception)

The paper does not say that all testing is useless. It says testing is impossible if you are trying to compare one "exposure map" against a broader, more complex "exposure map."

However, they offer a way out if you are willing to make extra assumptions that go beyond just the map.

The Analogy:
Instead of just guessing how people talk, you assume a specific mathematical rule: "People's behavior is a straight-line average of their neighbors' behavior." This is called a Linear-in-Means Model.

Because this rule is much stricter and more specific than a general "exposure map," the researchers were able to build a test that does work. It's like saying, "I don't know the exact rules of the game, but I'm willing to bet that the rules follow this specific, simple formula." If you are willing to make that specific bet, you can actually test if the data supports it.

Summary

The Bad News: You cannot use experimental data to prove that your general theory of "how people influence each other" (based on exposure maps) is correct. The data is too ambiguous; a test would be no better than guessing.
The Good News: If you are willing to restrict your theory to a very specific, mathematically simple shape (like a straight line), you can build a test that works.
The Lesson: Don't rely on data alone to validate your complex theories about social influence. You must bring in outside knowledge or make stronger assumptions to make the test work. The data alone is not enough to solve the mystery of the "perfect map."

Technical Summary: On the Impossibility of Specification Testing of Interference Models Based on Exposure Mappings

Problem Statement
In randomized experiments where spillovers (interference) are suspected, analysts must posit a model of interference to estimate causal effects. The dominant approach in empirical practice utilizes exposure mappings (Aronow & Samii, 2017), which reduce the high-dimensional treatment vector to a low-dimensional summary for each unit. A critical challenge arises because the true nature of spillovers is rarely known, leading researchers to employ specification tests to determine if a chosen interference model is correctly specified.

While existing literature has developed specification tests (often based on Fisher-style randomization tests) that control Type I error (rejecting a correct model), the Type II error (accepting an incorrect model) of these tests remains poorly understood. The central question addressed by this paper is whether it is possible to construct a specification test that can reliably distinguish between a correctly specified exposure mapping model and a misspecified one, particularly when the misspecification is "well-separated" from the null.

Methodology and Framework
The authors adopt a design-based framework for randomized experiments. They formalize the problem using the following components:

Models: The null model ( $M_0$ ) and alternative model ( $M_1$ ) are sets of potential outcome functions. Both are restricted by a uniform moment bound ( $\|y\|_\infty \le 1$ ) to ensure well-behaved outcomes.
Exposure Models: The paper focuses on models where the potential outcome of unit $i$ depends only on an exposure mapping $h_i(z)$ . A model is correctly specified if the outcome is invariant to changes in the treatment vector $z$ that do not change the exposure $h_i(z)$ .
Separation: To evaluate power, the authors define a separation functional $g(y)$ that measures the deviation of a potential outcome function from the null model. An alternative is $\delta$ -separated if $g(y) \ge \delta$ .
Performance Metric: The paper evaluates tests using the overall testing error $R(D, \phi, \delta)$ , defined as the sum of the worst-case Type I error and the worst-case Type II error for $\delta$ -separated alternatives.
Impossibility Criterion: A testing problem is deemed impossible if the minimax testing error $R^*(\delta) = 1$ . This implies that no testing procedure performs better than a naive procedure that discards all data and rejects the null at random.

Key Contributions and Results

Impossibility Theorem for Exposure Mappings:
The paper's primary result (Theorem 3.1) establishes a strong impossibility result: Specification testing of exposure mapping models is impossible.
- Specifically, for any null exposure model $M_0$ and any alternative exposure model $M_1$ (where $M_0 \subseteq M_1$ ), the minimax testing error is exactly 1 for all non-trivial separation values $\delta$ .
- This result holds for all sample sizes ( $n$ ), for uniformly bounded outcomes, and for alternatives that are maximally separated from the null.
- Proof Sketch: The authors construct explicit mixture distributions $\Pi_0$ (supported on $M_0$ ) and $\Pi_1$ (supported on the $\delta$ -separated $M_1$ ) such that the total variation distance between the distributions of observed outcomes is zero ( $d_{TV}(Q_0(z), Q_1(z)) = 0$ ) for any treatment assignment $z$ . Because the observed data distributions are indistinguishable under any experimental design, no test can achieve power better than random guessing.
Corollary on Sharp Nulls:
The impossibility extends to specific common cases, such as distinguishing between "no treatment effect" and "no interference" (SUTVA). Even if the treatment effect is maximally large (separated by the moment bound), the data distributions under the null and alternative mixtures are identical, rendering the models empirically indistinguishable.
Consistent Test for Non-Exposure Alternatives:
The authors demonstrate that the impossibility result is specific to the class of exposure mapping models. By restricting the alternative model to a structure that is not an exposure mapping, informative testing becomes possible.
- They construct a uniformly consistent test for the null hypothesis of no interference (SUTVA) against the alternative of a network linear-in-means model (where outcomes are linear in the fraction of treated neighbors).
- This test utilizes a consistent estimator of a quadratic functional of the potential outcomes under a Bernoulli design.
- The test is consistent provided the maximum degree of the network ( $d_{max}$ ) satisfies $d_{max} = o(n^{1/5})$ .

Significance and Claims
The paper claims that the inability to validate exposure mapping models using solely experimental data is a fundamental limitation, not a lack of methodological sophistication.

Epistemological Implication: The problem of model specification cannot be resolved through experimental data alone. The authors argue that researchers cannot rely on data to rule out all potential misspecifications of exposure models.
Practical Guidance: The authors assert that specification tests are only useful if researchers impose a priori restrictions on the alternative model that go beyond the structural constraints of exposure mappings. If an experimenter is unwilling to rule out any departures from the null a priori, the only valid conclusion is that the null may be misspecified.
Shift in Focus: The paper suggests that the field should move away from attempting to validate whether a specific exposure model is "correct" (a task shown to be impossible) and toward constructing tests that target specific, pre-specified types of departures (as illustrated by the linear-in-means example).
Role of External Data: The authors note that resolving model specification may require incorporating external data or qualitative methods, though this introduces new, potentially untestable assumptions.

In summary, the paper provides a formal proof that no specification test can uniformly distinguish between nested exposure mapping models, regardless of sample size or separation magnitude. Consequently, informative specification testing requires narrowing the scope of the alternative hypothesis beyond the class of exposure mappings.