Causal Effects with Unobserved Unit Types in Interacting Human-AI Systems

This paper proposes a causal message passing framework that enables the consistent estimation of human-specific causal effects in interacting human-AI systems by leveraging prior probabilities of unit types and analyzing outcome dynamics across subpopulations with varying expected human compositions, even when individual unit types and interaction networks remain unobserved.

The Gist

Imagine you are the manager of a massive, bustling online town square. In this square, real people are chatting, but there's a twist: you can't tell who is a real human and who is a robot.

The robots are so good at pretending to be human that they look, sound, and act almost exactly like people. However, you have a hunch (or a "prior") about each person: "There's a 70% chance this person is real, and a 30% chance they are a bot."

Now, imagine you want to test a new feature for the town square—let's say, posting a "Success Story" to make people happier. You want to know: Did this story make the real humans happier?

This is the problem the paper solves. Here is how they do it, explained simply.

The Problem: The "Noise" of the Robots

In a normal experiment, you'd just look at the people who saw the story and compare them to those who didn't. But here, the robots are messing things up in two ways:

1. They hide the signal: If the story makes humans happy but makes the cynical robots angry (and they stop posting), the average happiness of the whole town might look like "nothing changed." The good and bad cancel each other out.
2. They talk to each other: Humans and robots are all chatting in the same threads. If a robot gets angry and stops replying, a human might see that silence and stop replying too. This is called "network interference"—one person's reaction changes everyone else's.

Because you can't see who is who, and you can't see the invisible web of who is talking to whom, traditional math says, "You can't solve this."

The Solution: The "Grouping" Trick

The authors' brilliant idea is to stop looking at individuals and start looking at groups.

Think of it like sorting a bag of mixed red and blue marbles. You can't tell which specific marble is red or blue, but you do know that "Bag A" is 90% red and "Bag B" is 10% red.

The researchers do this with their users:

1. Sort by "Human-ness": They create different groups (subpopulations). Group A has mostly people who are likely human. Group B has mostly people who are likely bots. Group C is a mix.
2. Sort by "Exposure": Within those groups, they make sure some people saw the "Success Story" and some didn't.

Now, they have a set of groups that are different in two ways: how many humans are in them and how many saw the story.

The Magic: The "State Evolution" Recipe

Once they have these groups, they use a mathematical recipe (called Causal Message Passing or State Evolution).

Imagine you are a chef trying to figure out a secret ingredient in a soup. You can't taste the individual grains of salt, but you have three bowls of soup:

• Bowl 1: Mostly water, a little salt.
• Bowl 2: Mostly salt, a little water.
• Bowl 3: A perfect mix.

If you taste how the flavor changes as you add more salt to each bowl, you can mathematically reverse-engineer exactly how much salt is in the "pure salt" bowl, even if you never tasted the pure salt directly.

The researchers do the same thing with the data:

• They watch how the "Human-heavy" groups react to the story.
• They watch how the "Bot-heavy" groups react.
• They watch how the "Mixed" groups react.

Because the groups are different, the math can separate the "Human flavor" from the "Bot flavor." It calculates a trajectory (a path) of what would happen if everyone in the town square was a human and everyone saw the story, versus if no one saw it.

The Result: Seeing the Invisible

When they ran this on a simulation where AI bots were actually programmed to be cynical and humans were optimistic:

• The "Average" View: If you just looked at the whole town, the "Success Story" seemed to do nothing. The happy humans and angry bots canceled each other out.
• The "Human-Only" View: Their new method successfully ignored the bots and revealed the truth: The story made real humans 50% more engaged.

Why This Matters

This is a huge deal for the future of the internet. As AI agents become more common on social media, we are losing the ability to run fair experiments. We can't just ask, "Did this ad work?" because we don't know if the people clicking it are real or bots.

This paper gives us a mathematical microscope. It allows scientists and companies to run experiments and find out what is actually happening to real people, even when they are swimming in a sea of invisible robots. It turns a chaotic, unobservable mess into a clear, measurable signal.

Technical Summary

1. Problem Statement

The paper addresses the challenge of estimating human-specific causal effects in online platforms where human users interact with AI agents (bots), but the unit types (human vs. AI) and the interaction network structure are unobserved.

• Context: In modern digital ecosystems (e.g., social media), AI agents increasingly mimic human behavior, creating mixed populations. Interventions (treatments) affect both groups, but often with opposing or heterogeneous dynamics.
• The Core Difficulty:
  • Latent Types: Researchers cannot observe who is human and who is AI at the individual level. They only possess a prior probability $Q_i$ (e.g., from a classifier) that unit $i$ is human.
  • Network Interference: Outcomes depend on interactions between units (spillovers). Standard causal inference methods (like SUTVA) fail because the treatment of one unit affects the outcomes of others.
  • Identification Gap: Existing methods for network interference assume homogeneous units, while methods for heterogeneous treatment effects assume observed covariates. Neither handles the combination of unobserved types, unobserved networks, and dynamic interactions.
• Goal: To consistently estimate the Human Total Treatment Effect (H-TTE), defined as the difference in average human outcomes under full treatment vs. full control, despite the inability to distinguish individual unit types.

2. Methodology

The authors propose a framework based on Causal Message Passing (CMP) adapted for latent types. The approach shifts the analysis from individual-level data to aggregate subpopulation dynamics.

A. Theoretical Model

• Outcome Dynamics: The paper models the outcome $Y^i_t$ for unit $i$ at time $t$ as a function of:
  • Baseline heterogeneity ($\delta_H, \delta_A$) based on type.
  • Direct treatment effects ($\tau_H, \tau_A$) based on type.
  • Interference: Interactions with other units $j$ via weights $A_{ij}$ (fixed) and $B_{ij}^t$ (time-varying), which propagate treatments, lagged outcomes, and their interactions.
• Assumptions:
  • Type-Dependent Gaussian Interference: Interference weights are Gaussian random variables whose means depend on the type of the receiving unit ($U_i$).
  • Regularity: Standard concentration assumptions as population size $N \to \infty$.
  • Bernoulli Randomization: Treatments are assigned randomly.

B. Experimental State Evolution (ESE)

The key theoretical insight is that while individual types are hidden, the sample mean outcome of a sufficiently large subpopulation converges to a deterministic limit governed by the average human composition of that subpopulation.

• State Evolution Equation: The limit of the sample mean $\nu^S_t$ for a subpopulation $S$ with average human composition $q_S$ and treatment exposure $\pi^S_t$ is:
  $$\nu^S_t = \delta_H q_S + \delta_A(1-q_S) + (\tau_H q_S + \tau_A(1-q_S))\pi^S_t + (m_H q_S + m_A(1-q_S))(\dots)$$
  • Crucially, the complex high-dimensional network dynamics collapse into a low-dimensional recursion dependent only on the aggregate composition $q_S$.

C. Estimation Algorithm (Algorithm 1)

The estimation procedure involves three steps:

1. Subpopulation Construction: Construct $K$ subpopulations by stratifying units based on their priors $Q_i$ (to vary $q_S$) and their treatment histories (to vary $\pi_S$). This creates systematic variation in both composition and exposure.
2. Coefficient Estimation: Fit the State Evolution recursion to the observed aggregate trajectories of these subpopulations using non-linear least squares to estimate the structural parameters ($\theta = \{\delta, \tau, \alpha, \beta, \gamma\}$).
3. Counterfactual Projection: Use the fitted parameters to simulate two counterfactual worlds:
   • Full Treatment: All units treated, composition set to $q=1$ (pure human).
   • Full Control: No units treated, composition set to $q=1$.
   • The difference between these projected trajectories yields the estimated H-TTE.

3. Key Contributions

1. Theoretical Identification: The paper proves that distributional knowledge of population composition (the priors $Q_i$) is sufficient to identify human-specific causal effects, even without observing individual types or the interaction network.
2. Novel Framework: It extends the CMP framework (previously for homogeneous populations) to settings with latent heterogeneity, bridging the gap between network interference literature and latent variable models.
3. Consistency: The authors prove that the proposed estimator is consistent as $N \to \infty$, provided the experimental design includes sufficient variation in both composition and treatment exposure (Assumption 3).
4. LLM-Based Simulation: They developed a sophisticated simulator using Large Language Models (LLMs) with distinct prompting and temperature settings to create realistic, behaviorally differentiated human and AI agents, serving as a controlled testbed.

4. Experimental Results

The authors validated the method on a simulated dating platform with 200 users (50% human, 50% AI bots).

• Setup:
  • Humans: Optimistic, engaged by positive "success stories."
  • AI Bots: Cynical, disengaged by positive content (negative response to treatment).
  • Treatment: Exposure to "Success Stories."
• Findings:
  • Ground Truth: The treatment had a strong positive effect on humans (+0.5) but a negative effect on AI bots (-0.4). The population average effect was near zero (+0.04) due to cancellation.
  • Performance: Algorithm 1 accurately recovered the human-specific effect (H-TTE $\approx$ 0.5), closely tracking the ground truth.
  • Baselines:
    • Standard Difference-in-Means (DIM) and Hájek estimators failed, predicting near-zero effects.
    • Existing CMP methods (designed for homogeneous populations) also failed to isolate the human effect, estimating the population average instead.
  • Robustness: The method remained accurate even with moderate classifier noise (prior accuracy $a=0.7$ to $0.9$), though performance degraded slightly as priors became less informative.

5. Significance

• Practical Relevance: As AI agents proliferate on digital platforms, traditional A/B testing becomes unreliable because "noise" from bots can mask or distort human responses. This framework provides a rigorous way to isolate human impact.
• Methodological Advance: It demonstrates that aggregate data combined with probabilistic priors can solve identification problems that are impossible with individual-level data when types are hidden.
• Policy & Design: This enables platform designers and policymakers to evaluate interventions (e.g., content moderation, algorithmic changes) specifically for human well-being, ensuring that AI-driven dynamics do not obscure the true effects on human users.

In summary, the paper establishes that by leveraging the statistical properties of large populations and the known distribution of unit types, one can "de-noise" the interference of AI agents to recover the true causal impact of interventions on human users.