Estimating Treatment Effects under Algorithmic Interference: A Structured Neural Networks Approach

This paper proposes a structured semiparametric framework combining an algorithm choice model and a viewer response model to correct the severe bias in standard estimators caused by algorithmic interference in two-sided marketplaces, thereby enabling accurate global treatment effect estimation for platform-wide algorithm updates.

The Gist

Imagine you are the manager of a massive, bustling digital town square (like TikTok or YouTube). In this square, thousands of street performers (creators) are trying to get the attention of passersby (viewers). To help the best performers get noticed, you have a "Traffic Director" (the algorithm) who decides who gets to stand on the main stage and who gets pushed to the back.

Every few months, you want to test a new Traffic Director to see if it does a better job than the old one. But here's the problem: You can't just watch the new director work in isolation.

The Problem: The "Crowded Stage" Effect

In the real world, when you test a new director, you can't give them their own private stage. You have to mix the new director's performers with the old director's performers on the same stage.

This creates a tricky situation called Algorithmic Interference.

Think of it like a game of musical chairs, but the chairs are "viewer attention."

• If your new director is really good at spotting talent, they might push their performers to the front of the line.
• Because there are only a few spots on the main stage, those new performers push out the old performers.
• Suddenly, the old performers get fewer chances to perform, not because they are bad, but because the new director is too good at picking winners.

If you just count who got the most applause (likes/views) at the end of the day, you might get the wrong answer:

1. The "Crowding Out" Bias: You might think the new director is amazing because their performers got more applause. But really, you just stole the spotlight from the old performers.
2. The "Wrong Crowd" Bias: The new director might be great at finding performers who appeal to boring people, while the old director appealed to exciting people. If you only look at the applause, you might miss that the new director is actually annoying the wrong audience.

The Result: The standard way of measuring success (just comparing the two groups) is like judging a race where the runners are tripping over each other. You might declare a winner who actually lost, or a loser who actually won. In the real world, this could cost a company billions of dollars by rolling out a bad algorithm.

The Solution: The "Crystal Ball" Simulator

The authors of this paper (Ruohan Zhan, Shichao Han, et al.) built a smart simulator to fix this. Instead of just watching the race and guessing, they built a model that understands how the race works.

They used two main tools, like a detective solving a mystery:

1. The "Choice Model" (The Traffic Director's Brain):
   They built a neural network (a type of AI) to figure out exactly how the algorithm decides who gets on stage. It learns: "When a viewer likes cats, and Creator A is a cat video, Creator A gets a high score. But if Creator B is also a cat video, the score changes."
   This model understands the competition. It knows that if Creator A gets a boost, Creator B might get pushed down.

2. The "Response Model" (The Audience's Heart):
   They built another AI to predict how viewers react once they actually see a video. "If a viewer sees a cat video, how likely are they to laugh?"

The Magic Trick (Debiasing):
Once they have these two models, they don't just look at the real-world data. They run a simulation in their computer:

• Scenario A: What if everyone used the new director? (The AI simulates the new director picking winners, pushing the old ones out, and predicts the total applause).
• Scenario B: What if everyone used the old director? (The AI simulates the old director picking winners).

By comparing these two simulated worlds, they can see the true effect of the new algorithm, completely ignoring the messy interference of the mixed experiment.

They call this a "Debiased Estimator." It's like having a pair of glasses that filters out the noise of the crowd so you can see the true performance of the runners.

Why This Matters (The "Double-Sided" Cost)

Usually, to get the perfect answer, companies try to do a "Double-Sided Experiment." This is like building two separate, parallel universes:

• Universe 1: Only the new director and their performers.
• Universe 2: Only the old director and their performers.

This gives a perfect answer, but it's expensive and dangerous. You cut your audience in half, so the market feels "thin" (less variety), and you need twice the engineering work. It's like testing a new car engine by building two separate garages instead of just testing it on the road.

The authors' method is brilliant because it lets you test the new engine on the same road (the standard experiment) but uses their "Crystal Ball" math to tell you what would have happened if you had built two separate garages.

The Real-World Test

The team tested this on Weixin Channels (a huge short-video platform in China).

• They ran a standard experiment mixing old and new algorithms.
• They also ran the expensive "two universes" test just to see the truth.
• The Result: The standard methods (just counting likes) said the new algorithm was great. The "Two Universes" test said the new algorithm was actually terrible (it hurt the platform).
• The Winner: The authors' new "Debiased Estimator" correctly predicted that the new algorithm was terrible, matching the expensive "Two Universes" test.

The Takeaway

In a world where algorithms are constantly fighting for our attention, simple math isn't enough. You can't just compare Group A and Group B because they are influencing each other.

This paper gives companies a smart, mathematical telescope that lets them see the true impact of their changes without having to build expensive, parallel worlds. It saves money, prevents bad decisions, and ensures that the "Traffic Director" we choose actually helps everyone, not just a lucky few.

Technical Summary

Here is a detailed technical summary of the paper "Estimating Treatment Effects under Algorithmic Interference: A Structured Neural Networks Approach" by Zhan, Han, Hu, and Jiang.

1. Problem Statement

Context: Online content platforms (e.g., short-video apps) use algorithms to allocate promotional traffic to creators. To evaluate new algorithm updates, platforms typically run creator-side randomized experiments, where creators are randomly assigned to treatment (new algorithm) or control (old algorithm) groups.

The Core Issue: Algorithmic Interference
Standard causal inference relies on the Stable Unit Treatment Value Assumption (SUTVA), which assumes a unit's outcome depends only on its own treatment. However, in two-sided marketplaces, treated and control creators compete for the same viewer impressions within a consideration set.

• Mechanism: If the treatment algorithm boosts scores for treated items, they "crowd out" control items, altering the exposure distribution.
• Consequence: The observed outcome of a creator depends not only on their own treatment status but also on the treatment status of their competitors. This violates SUTVA, leading to algorithmic interference.
• Bias Sources:
  1. Content Exposure Bias: The realized share of exposed treated items differs from the random assignment probability (e.g., treated items get 56% of exposure despite 50% assignment).
  2. Viewer Selection Bias: The algorithm's personalization means treated items may be shown to a systematically different audience (e.g., high-value viewers) compared to control items.

The Goal: Estimate the Global Treatment Effect (GTE)—the impact of rolling out the new algorithm to all creators—using data from a standard creator-side experiment, avoiding the prohibitive cost of "double-sided" experiments (which partition the market to eliminate interference).

2. Methodology

The authors propose a Structured Semiparametric Framework combined with Double/Debiased Machine Learning (DML).

A. Modeling the Interference Pathway

The approach decomposes the data-generating process into two distinct models:

1. Algorithm Choice Model (Exposure Mechanism):

   • Models how exposure is allocated among competing items in a consideration set.
   • Structure: A semi-parametric Multinomial Logit model where the latent score $S_{i,k}$ for item $k$ and viewer $i$ is:
     $$S_{i,k} = s_0(V_i, C_{i,k}) + W_{i,k} \cdot s_1(V_i, C_{i,k}) + \epsilon_{i,k}$$
     • $s_0$: Baseline score (control algorithm).
     • $s_1$: Treatment uplift (difference introduced by the new algorithm).
     • Neural Networks: Both $s_0$ and $s_1$ are flexible neural networks parameterized to capture complex, high-dimensional interactions between viewer and content features.
   • Interference Capture: The exposure probability $p_k$ depends on the scores of all items in the set, explicitly modeling the competition.

2. Viewer Response Model (Outcome Mechanism):

   • Models the viewer's engagement (e.g., like, watch time) conditional on exposure.
   • Structure: $Y_i = z(V_i, C_{i,k}) + \zeta_i$.
   • Neural Networks: A flexible neural network $z(\cdot)$ predicts the outcome.
   • Assumption: Conditional on exposure, the outcome does not depend on the treatment status (viewers are unaware of the algorithm change).

B. Counterfactual Estimation

To estimate the GTE, the authors simulate two counterfactual worlds using the estimated models:

1. Global Treatment ($\pi_1$): All items in the consideration set are assigned $W=1$.
2. Global Control ($\pi_0$): All items are assigned $W=0$.
   The GTE is the difference in expected outcomes between these two simulated policies.

C. Debiased Estimator (DML Framework)

Since the nuisance functions ($s_0, s_1, z$) are estimated via neural networks (which converge slower than $\sqrt{n}$), a naive "plug-in" estimator would be biased. The authors construct a Debiased (DB) Estimator:

• Neyman Orthogonality: They construct an estimator $\psi$ that is locally robust to first-order errors in the nuisance function estimation.
• Correction Term: The estimator adjusts the plug-in prediction using gradients of the loss functions and the Hessian matrix:
  $$\psi^{DB} = \mu(\hat{\theta}) - \nabla\mu(\hat{\theta})^T H^{-1} \nabla\ell(\hat{\theta})$$
• Cross-Fitting: To avoid overfitting, the data is split into folds; nuisance models are trained on $K-1$ folds and evaluated on the held-out fold.

D. Theoretical Extension: Correlated Samples

A key methodological innovation is extending DML asymptotic theory to correlated samples.

• Problem: In creator experiments, items appear in multiple overlapping consideration sets. Once an item's treatment status is fixed, its status in overlapping sets is deterministic, violating the i.i.d. assumption standard in DML.
• Solution: The authors model the data sequentially and use martingale limit theorems. They prove that under mild conditions (items do not dominate the market), the debiased estimator remains $\sqrt{n}$-consistent and asymptotically normal despite sample correlation.

3. Key Contributions

1. Substantive: Provides a reliable method to evaluate promotional algorithms using standard, low-cost creator-side experiments, avoiding the need for expensive double-sided market partitioning.
2. Methodological:
   • Introduces a Structured Neural Network framework that combines the interpretability of choice models with the flexibility of deep learning.
   • Extends Double Machine Learning (DML) theory to handle correlated data arising from overlapping consideration sets, a common feature in marketplace and panel data.
   • Demonstrates that standard estimators (Difference-in-Means, Propensity Weighting) fail catastrophically under algorithmic interference.

4. Results

A. Monte Carlo Simulations

• Setup: Simulated environments with varying consideration set sizes ($K=3, 8$).
• Findings:
  • Debiased (DB) Estimator: Unbiased and provides valid confidence intervals.
  • Difference-in-Means (DIM): Severely biased (up to 400% error in some cases) and often predicts the wrong sign of the effect.
  • Pure Deep Learning (PDL): Biased due to poor extrapolation to counterfactual global policies.
  • Propensity-based (IPW/AIPW): Theoretically unbiased but suffer from exponential variance as the consideration set size ($K$) increases, rendering them unstable even for moderate $K$.

B. Empirical Application (Weixin Channels)

• Design: A large-scale field experiment on Tencent's Weixin short-video platform.
  • Benchmark: A costly "double-sided" experiment (partitioning viewers/creators) served as the interference-free ground truth.
  • Test: Standard creator-side experiment data was analyzed using the proposed DB estimator and benchmarks.
• Evidence of Bias:
  • Exposure Bias: Treated items received 56% of exposures despite 50% assignment.
  • Selection Bias: Significant systematic differences in viewer demographics between treated and control exposures.
• Performance:
  • Outcome 2 (Critical Case): The ground truth showed a significant negative effect (the new algorithm was worse).
    • DB Estimator: Correctly identified the negative effect.
    • DIM Estimators: Incorrectly reported a significant positive effect.
    • PDL: Wildly inaccurate.
  • Outcome 1: Ground truth was null; DB correctly found no effect, while DIM found a false positive.
  • Conclusion: Relying on standard estimators would have led the platform to deploy a harmful algorithm. The DB estimator consistently matched the ground truth.

5. Significance

• Practical Impact: Offers platforms a cost-effective, statistically rigorous tool to make high-stakes algorithm deployment decisions without the massive resource overhead of double-sided experiments.
• Theoretical Advancement: Bridges the gap between structural econometrics (choice models) and modern machine learning (neural networks) while solving a critical theoretical gap in causal inference regarding interference in correlated data.
• Business Relevance: Highlights the danger of "naive" A/B testing in competitive digital marketplaces, where interference can reverse the apparent direction of treatment effects, leading to suboptimal or damaging business strategies.