🤖 machine learning

Actions Have Consequences: Detecting Outcome Performativity using Intervention Testing

This paper introduces Outcome Performativity A/B Detection (OPAB), a formal framework that uses intervention testing to detect when predictions causally influence their own outcomes, while deriving and validating sample complexity bounds to identify settings where such detection is feasible or impossible due to data constraints.

Original authors: Brandon Gower-Winter, Georg Krempl

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Brandon Gower-Winter, Georg Krempl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster. Usually, you just look at the clouds and say, "It's going to rain." Your prediction doesn't change the weather; the sky does what it wants regardless of what you say. But what if your prediction could change the weather? What if, the moment you shouted "Rain!" to the crowd, the clouds actually gathered because people started opening umbrellas, and the wind shifted? In the world of Artificial Intelligence, this strange phenomenon is real. It's called Outcome Performativity. It happens when a computer's guess about the future actually helps create that future. Think of a bank's AI predicting who will default on a loan. If the AI says, "This person is risky," and the bank denies them a loan, that person might actually go bankrupt because they were denied the money. The prediction caused the outcome.

This is a tricky problem for scientists because it creates a feedback loop. If an AI learns from data where its own predictions changed the results, it might get confused, thinking it's a genius when it's actually just a self-fulfilling prophet. The big question is: How do we catch this behavior before we let the AI loose on the real world? We need a way to test if our predictions are secretly pulling the strings. This is where a new method called OPAB (Outcome Performativity A/B Detection) comes in, acting like a scientific detective to see if the AI is secretly controlling the game.

The Detective's Toolkit: OPAB

The paper introduces a method called OPAB, which is essentially a "what-if" experiment for AI predictions. To understand how it works, imagine you are running a school cafeteria. You want to know if telling students "The pizza is bad" actually makes them eat less pizza.

In a normal experiment, you might just ask the students what they think of the pizza and see how much they eat. But that's flawed because the students who think the pizza is bad might already be picky eaters. The prediction (their opinion) and the outcome (eating) are tangled together.

OPAB untangles this knot by using a technique called Intervention Testing. Instead of letting the students decide what they think, the cafeteria manager flips a coin for every student. Heads, you tell them, "The pizza is amazing!" Tails, you tell them, "The pizza is terrible!" You force the prediction, completely ignoring what the student actually thinks or who they are. Then, you watch what happens. If the group told "The pizza is amazing" eats significantly more than the group told "The pizza is terrible," you have caught the AI (or the cafeteria manager) in the act of Outcome Performativity. The prediction caused the change in behavior.

The Rules of the Game: When Can We Catch It?

The authors didn't just build the detector; they also figured out exactly how many students (or data points) you need to run this experiment to be sure you aren't just seeing a fluke. They call this Sample Complexity.

Think of it like trying to hear a whisper in a noisy room. If the whisper is very loud (a strong effect), you only need a few people to hear it. But if the whisper is barely audible (a tiny effect), you might need a stadium full of people to be sure you heard it and didn't just imagine it.

The paper explores three different "scenarios" or rules for how the AI might be influencing the world:

The Simple Rule: The prediction changes the outcome directly, like a magic switch.
The Model Rule: The prediction changes the outcome based on the specific details of the person (like their age or location).
The Mistake Rule: The prediction only changes the outcome if the AI makes a mistake (like telling a healthy person they are sick).

For each of these, the authors derived mathematical formulas to calculate the minimum number of "coin flips" (interventions) needed. They found that if the effect is very small, the number of people you need to test shoots up to infinity. This creates what they call Regions of Indistinguishability. These are settings where the effect is so subtle, or the cost of testing is so high, that it is practically impossible to tell if the AI is performing or not. It's like trying to hear a pin drop in a hurricane; no matter how many people you ask, you can't be sure.

The Real-World Test: The Fashion Store

To see if their theory held up, the authors tested OPAB on a real dataset from a fashion website called ZOZOTOWN. This dataset tracked what clothing items were shown to users and whether they clicked on them. Crucially, for some users, the position of the item (left, center, or right) was chosen randomly, just like the coin flip in our cafeteria story.

They split the data into men and women. When they applied OPAB to the men's data, the detector went off: the position of the item did change whether people clicked (Outcome Performativity). But when they tested the women's data, the detector stayed silent. This suggests that for women, the order of items didn't matter, but for men, it did. This proves that OPAB can spot these hidden influences in the real world.

The Limits and the Future

The paper is careful to point out that this isn't a magic bullet that solves everything. The method works best when you have enough data. If the effect is tiny, or if the data is very unbalanced (like having way more clicks on one item than another), the detector might miss it. The authors also note that this method is an offline tool, meaning it's best used before you launch a new AI system, during the testing phase, rather than trying to fix it after it's already running.

They compared their method to an older, online method that waits for the AI to run and then checks if the results changed. They found that their "coin flip" method (OPAB) was much more efficient and reliable, catching the problem with far fewer data points.

In the end, this paper gives us a new way to peek behind the curtain. It tells us that while we can't always catch every subtle influence an AI might have, we have a mathematical map that shows us exactly where the danger zones are and how much data we need to be safe. It's a crucial step toward making sure our AI tools help us rather than accidentally tricking us into creating the very problems they were supposed to solve.

Technical Summary: Detecting Outcome Performativity Using Intervention Testing

Problem Definition

In many critical domains such as Palliative Care, Credit Assignment, and Recommender Systems, the predictions generated by Machine Learning (ML) models can causally influence the outcomes they are intended to predict. This phenomenon, known as Outcome Performativity, occurs when the realized outcome $Y$ of an instance $X$ depends not only on the features $X$ but also on the prediction $\hat{Y}$ assigned to that instance. For example, assigning high interest rates to high-risk debtors may increase the likelihood of default, or assigning palliative care instead of curative care may influence a patient's survival.

While existing research on Performative Prediction often assumes the presence of such feedback loops is known a priori, this paper addresses the gap in detecting Outcome Performativity, particularly in offline settings (during data collection or before model deployment). Undetected performativity can lead to models that exacerbate bias or suffer from performance decay. Current online methods for identifying these effects exist, but there is limited work on identifying them offline, which is crucial for ethical and financial reasons before a model is deployed.

Methodology: Outcome Performativity A/B Detection (OPAB)

The authors propose Outcome Performativity A/B Detection (OPAB), an offline intervention testing method inspired by Randomized Controlled Trials (RCTs) and A/B testing.

Core Mechanism

Standard hypothesis testing comparing outcome distributions conditioned on a classifier's predictions ( $P(Y|\hat{Y}=0)$ vs. $P(Y|\hat{Y}=1)$ ) is insufficient because the predictions $\hat{Y}$ themselves are dependent on the features $X$ , creating confounding variables. To break this dependency, OPAB performs a do-intervention on the predictions.

Random Assignment: Instead of using a trained predictor $\phi(X)$ , OPAB randomly assigns predictions $\hat{y} \in \{0, 1\}$ to instances $X$ with equal probability (Bernoulli(0.5)). This decouples $\hat{Y}$ from $X$ .
Observation: The system observes the resulting outcomes $Y$ for these randomly assigned predictions.
Hypothesis Testing: OPAB statistically tests the dissimilarity between the outcome distributions of the two intervention groups:
$H(P(Y | \text{do}(\hat{Y}=0)), P(Y | \text{do}(\hat{Y}=1))) \leq \delta$
If the distributions are significantly different (where $H$ is a test like Chi-Squared or Fisher's Exact, and $\delta$ is the significance threshold), Outcome Performativity is detected.

Assumption Classes

The paper derives sample complexity bounds (the minimum number of interventions $N$ required) under three distinct assumption classes of Outcome Performativity:

Simple Outcome Performativity: The outcome probability depends only on the prediction $\hat{y}$ , abstracting away the feature space $X$ .
Model-based Outcome Performativity: The outcome depends on the instance's position in the feature space and the prediction, modeled as a linear shift in feature space based on prediction weights.
Misclassification-based Outcome Performativity: Outcome Performativity only occurs when the prediction $\hat{y}$ mismatches a "desirable" outcome $\bar{y}$ . The probability of the desired outcome being realized despite misclassification is governed by parameters $\gamma_0$ and $\gamma_1$ .

Key Contributions

Formalization of Offline Detection: The paper formalizes the use of intervention testing (OPAB) to detect Outcome Performativity prior to model deployment, contrasting with methods that require a deployed model to observe feedback loops.
Sample Complexity Bounds: The authors derive theoretical bounds for the number of samples ( $N$ ) required to detect Outcome Performativity for the three assumption classes. These bounds reveal that detection becomes increasingly difficult (requiring $N \to \infty$ ) as the difference in performative response between prediction groups approaches zero.
Regions of Indistinguishability: A key concept introduced is the "region of indistinguishability." These are parameter settings where the allotted number of interventions is insufficient to reliably detect Outcome Performativity. This highlights practical limitations in scenarios where samples are scarce, costly, or ethically restricted.
Empirical Validation: The method is validated on synthetic data imputed with performativity across six real-world binary classification datasets (Breast Cancer, Diabetes, Adult Census, Kickstarter, Titanic, Loan) and a real-world case study on the Open Bandits Dataset (OBD).

Results

Detection Efficacy: OPAB successfully detects Outcome Performativity in various settings. Detection rates increase with the number of interventions ( $N$ ).
Indistinguishability: The experiments confirm the existence of regions of indistinguishability. For instance, in Simple Outcome Performativity, if the performative response parameters $\alpha_0$ and $\alpha_1$ are nearly identical, even large sample sizes ( $N=10,000$ ) fail to detect performativity.
Dataset Specifics:
- In the Open Bandits Dataset, OPAB detected Outcome Performativity in the "men" sub-dataset (where item placement influenced clicks) but not in the "women" sub-dataset.
- Class Imbalance: Results indicate that class imbalance significantly impacts detection. Balanced datasets allow for more efficient detection, while highly imbalanced datasets (e.g., the OBD click-through rates) require exponentially more samples to detect subtle effects.
Comparison with Online Methods: A comparison with an online detection method (Adam Detection, based on monitoring True Positive Rate shifts) shows that OPAB achieves higher and more consistent detection rates using only 10% of the samples required by the online method. OPAB is also immune to concept drift issues that affect online methods.

Significance and Claims

The paper claims that OPAB provides a practical, sample-efficient tool for practitioners to identify Outcome Performativity before deploying models, potentially preventing the deployment of systems that inadvertently harm outcomes or reinforce bias.

The derivation of sample complexity bounds offers a critical insight: there are settings where detecting Outcome Performativity is practically infeasible due to the prohibitive number of interventions required. This has significant implications for high-stakes domains (like healthcare) where intervention testing may be unethical or too expensive. In such cases, if a setting falls within a "region of indistinguishability," practitioners may need to rely on non-interventional methods (which are noted to be less sample-efficient) or accept the risk of undetected performativity.

The authors conclude that while OPAB is effective, it relies on the assumption of uniformity across the feature space and does not account for verification latency or noisy labels. Future work aims to address these limitations.