Policy relevance of causal quantities in networks

This paper argues that while many common causal estimands in network settings fail to simultaneously offer interpretable summaries of unit-level effects and support optimal policy choice due to the mismatch between homogeneous exposure assumptions and heterogeneous realities, the expected average outcome serves as a superior estimand that fulfills both criteria.

The Gist

Imagine you are the mayor of a town, and you want to decide whether to launch a new public health campaign (like a free flu shot or a new recycling program).

In a simple world, you just look at the people who got the shot and compare them to those who didn't. If the vaccinated group is healthier, you give the shot to everyone. This is the standard way scientists measure "cause and effect."

But here's the problem: People don't live in bubbles. If your neighbor gets the flu shot, you are less likely to get the flu, even if you didn't get one. This is called interference or spillover. Your outcome depends not just on your own treatment, but on what your neighbors do.

This paper argues that when we have these "network effects," most of the statistical tools researchers use to measure success are actually misleading guides for policy. They might tell you what happened, but they won't tell you what to do next.

Here is the breakdown using a simple analogy: The "Party Planner" Dilemma.

The Two Ways to Measure a Party's Success

Imagine you are planning a party. You want to know: "How happy will the guests be?"

Method A: The "Local Vibe" Check (What researchers usually do)

You ask every guest: "How happy are you based on how many of your friends are here?"

• You calculate the average happiness of guests who have 0 friends at the party.
• You calculate the average happiness of guests who have 1 friend at the party.
• You calculate the average happiness of guests who have 2+ friends at the party.

The Trap: This tells you that "Guests with 1 friend are the happiest."
The Policy Failure: You now think, "Great! I should invite guests until everyone has exactly one friend."
The Reality: In a real network (like a town or a social media graph), you often cannot arrange the party so that everyone has exactly one friend. Some people will inevitably have 0, some will have 3, and some will have 5.
Because you only looked at the "Local Vibe" (Method A), you don't know what the total happiness of the whole party will be under a realistic invitation list. You might invite 50% of the town, thinking it's perfect, but end up with a chaotic mix of lonely people and overcrowded groups, resulting in a terrible party.

Method B: The "Total Party Score" (What the authors suggest)

Instead of asking "How happy are people with X friends?", you ask: "If I invite 50% of the town randomly, what is the total average happiness of the entire party?"

You simulate the whole party, look at the final result, and then try a different percentage (say, 60%) and see the new total result.

The Benefit: This number (called the Expected Average Outcome or EAO) tells you exactly what will happen if you choose a specific policy. It doesn't matter if the distribution of friends is messy; it just tells you the bottom line: "If we do Policy X, the town's total well-being will be Y."

The Core Argument

The authors, Sahil Loomba and Dean Eckles, argue that the scientific community has been obsessed with Method A (the "Local Vibe" or AFEO).

• Why they like it: It feels very "scientific" and causal. It isolates specific effects (e.g., "Having one treated neighbor causes a 5% boost").
• Why it fails for policy: It assumes you can control the environment perfectly to create those specific conditions. In the real world, you can't force everyone to have exactly one treated neighbor. So, these numbers are often useless for deciding which policy to actually implement.

They argue we should switch to Method B (the EAO).

• Why it works: It averages over both the people and the random chance of who gets treated. It answers the question the decision-maker actually cares about: "Which policy makes the whole population better off?"
• The Twist: Even though Method B is less "pure" in terms of isolating specific causal mechanisms, it is the only number that is guaranteed to help you pick the best policy, especially when you care about the total well-being of the group (utilitarian welfare).

The "Map" Metaphor

Think of the network of people as a city map.

• Method A gives you a detailed report on every single street corner: "Corner A has 2 potholes, Corner B has 1."
• Method B gives you a traffic report: "If we send 10% of the cars down Main Street, the average commute time for the whole city will be 20 minutes."

If you are a city planner trying to reduce traffic, the street-corner report (Method A) is interesting, but it doesn't tell you how to set the traffic lights to minimize the total commute time. You need the traffic report (Method B).

The Takeaway

1. Stop getting distracted by the details: Just because a statistical number sounds like a precise "causal effect" (like "the effect of having one treated neighbor") doesn't mean it helps you make a decision.
2. Focus on the bottom line: If you want to know which policy to choose, you need to estimate the Expected Average Outcome (EAO). This is the average result you get if you run the whole policy on the whole population.
3. The "Double Average": The authors show that if you average the results correctly (averaging over people and over the randomness of the policy), you get a number that is both easy to interpret and perfectly suited for making decisions.

In short: Don't try to optimize for a hypothetical scenario where everyone has the exact same number of treated neighbors. Instead, calculate the average outcome of the messy, real-world policies you can actually implement. That is the only number that will save you from making a bad decision.

Technical Summary

Here is a detailed technical summary of the paper "Policy relevance of causal quantities in networks" by Sahil Loomba and Dean Eckles.

1. Problem Statement

In causal inference settings involving interference (e.g., social networks), the outcomes of units depend not only on their own treatment but also on the treatments of their neighbors. This "interference" complicates the definition of causal effects. While the Average Treatment Effect (ATE) is the standard estimand in non-interference settings, the proliferation of proposed estimands in network settings has created ambiguity regarding which quantities are most useful.

The authors identify a critical gap: many commonly used estimands fail to satisfy two simultaneous desiderata required for effective policy-making:

1. Unit-Level Causal Interpretability: The estimand should be interpretable as a summary of unit-level causal effects (e.g., an average of how a specific change in exposure affects an individual).
2. Policy Relevance (Sufficiency): The estimand should be sufficient to guide the choice of an optimal policy (e.g., determining the treatment probability $\pi$ to maximize utilitarian welfare).

The paper argues that most existing estimands satisfy one property but fail the other, leading to potential missteps in policy evaluation.

2. Methodology and Framework

The authors formalize the problem using a framework of double averaging over two dimensions:

• Units ($i$): The population of $n$ units.
• Treatment Assignments ($Z$): The vector of treatments under a specific policy $\pi$.

They categorize estimands based on the order of averaging:

A. Averaging over Assignments, then Units (AFEO)

• Definition: First, calculate the expected outcome for a unit $i$ conditional on a specific exposure (e.g., having exactly $k$ treated neighbors), then average these expectations across all units.
• Formal Name: Average Focal Expected Outcome (AFEO).
• Common Usage: This is the standard approach in literature (e.g., Aronow & Samii, 2017; Savje et al., 2021). It produces "dose-response curves" based on exposure levels (e.g., outcome vs. number of treated neighbors).
• Analysis:
  • Interpretability: Under a correctly specified exposure map, AFEOs are interpretable as averages of unit-level causal effects.
  • Policy Relevance: Insufficient. The authors prove that AFEOs are only sufficient for optimal policy choice if the policy induces a unit-homogeneous exposure distribution (i.e., every unit has the same probability distribution over exposures). In irregular networks, even simple homogeneous policies (like Bernoulli trials) create heterogeneous exposures. Consequently, AFEOs cannot uniquely identify the Expected Average Outcome (EAO) under a policy, making them poor guides for optimization.

B. Averaging over Units, then Assignments (EFAO)

• Definition: First, calculate the average outcome of a specific subset of units (focal units) for a specific treatment assignment $Z$, then take the expectation of this average over the distribution of $Z$ defined by policy $\pi$.
• Formal Name: Expected Focal Average Outcome (EFAO).
• Special Case: When the focal set includes all units, this is the Expected Average Outcome (EAO).
• Analysis:
  • Policy Relevance: Sufficient. The EAO directly represents the expected welfare under a policy. Contrasts in EAO between policies are sufficient for choosing the optimal policy under utilitarian welfare.
  • Interpretability: Generally, EFAO contrasts are not unit-level causal effects. They represent the causal effect of a policy on aggregate outcomes, not necessarily an average of individual causal mechanisms. However, the authors argue this is still a valid causal quantity for decision-making.

3. Key Contributions and Results

1. The Commutativity of Averaging

The paper establishes that the two averaging orders (Units $\to$ Assignments vs. Assignments $\to$ Units) generally do not commute.

• Proposition 3: The only focal map where these two orders always coincide (universally commuting) is the deterministic focal map (specifically, the map including all units).
• Result: The Expected Average Outcome (EAO) is the unique estimand where the order of averaging does not matter. It is simultaneously a summary of unit-level effects (in a broad sense) and sufficient for policy choice.

2. Limitations of AFEOs (The "Compression" Problem)

The authors demonstrate that AFEOs "compress" the potential outcome table in a way that discards information necessary for policy optimization.

• Identification Failure: Even with a correctly specified exposure map, AFEOs only partially identify the EAO unless the exposure distribution is unit-homogeneous.
• Empirical Illustration: Re-analyzing data from Cai et al. (2015) on insurance adoption, the authors show that while AFEOs suggest a specific "optimal" exposure level (e.g., 1 treated neighbor), no simple policy can induce this exposure for all units simultaneously. The bounds on the true welfare derived from AFEOs are often too wide to inform a decision.

3. Re-evaluating Causal Interpretation

The paper challenges the dogma that an estimand must be a "unit-level causal effect" to be useful.

• Argument: While EFAO contrasts (including EAO) are not always decomposable into unit-level contrasts, they are causal effects of the policy on aggregate outcomes. For a decision-maker maximizing utilitarian welfare, the aggregate outcome is the primary object of interest.
• Conclusion: Researchers should prioritize estimands that are sufficient for policy choice (EAO) over those that are merely interpretable as unit-level effects but insufficient for decision-making (AFEO).

4. Formal Characterization of Sufficiency

• Proposition 10: A set of estimands is universally sufficient for policy choice under utilitarian welfare (with arbitrary costs) if and only if it functionally identifies the EAO (up to a constant).
• Corollary 11.2: AFEOs identify the EAO if and only if the exposure distribution under the policy is unit-homogeneous.

4. Significance and Implications

• For Methodologists: The paper provides a rigorous mathematical characterization of why many popular network estimands fail in policy contexts. It advocates for shifting focus toward the Expected Average Outcome (EAO) and its contrasts as the primary target for estimation in network experiments.
• For Empirical Researchers: It warns against the intuitive but often incorrect practice of using "exposure-response curves" (AFEOs) to directly select optimal treatment probabilities. Instead, researchers should estimate the expected welfare under specific policy scenarios (EAOs).
• For Policy Makers: The findings suggest that in networked environments, the "optimal dose" is not a static number of neighbors treated, but a specific policy distribution that accounts for network heterogeneity. The paper argues that policy-relevant quantities must reflect the aggregate outcome of the policy, even if that quantity is harder to decompose into individual causal mechanisms.

Summary of the Core Argument

The paper posits a trade-off:

1. AFEOs are interpretable as unit-level effects but are insufficient for policy choice because they ignore the heterogeneity of exposure induced by policies.
2. EFAOs (specifically EAO) are sufficient for policy choice but are often not interpretable as simple unit-level effects.
3. Resolution: The EAO is the unique estimand that satisfies both criteria (it is the intersection of the two averaging orders). Therefore, the research community should prioritize the estimation of EAOs to ensure causal inference directly informs optimal policy design.