📊 statistics

Causal Inference with Unstructured Outcomes

This paper proposes a novel causal inference framework for unstructured outcomes, such as text and images, by introducing the "maximally contrasting feature" (MCF) to identify and estimate the specific aspects of complex data most significantly altered by a treatment, overcoming the limitations of traditional scalar-based average treatment effects.

Original authors: Kevin Christian Wibisono, Yixin Wang

Published 2026-08-05

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Kevin Christian Wibisono, Yixin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out why a crime happened. Usually, the clues are simple numbers: how much money was stolen, how many people were hurt, or how long a suspect waited before confessing. You can easily compare these numbers to see if a new police tactic made a difference. But what if the "clue" isn't a number at all? What if the evidence is a messy, handwritten note, a blurry photo, or a long, rambling story? You can't subtract one story from another to find the difference, just like you can't subtract one painting from another to see how an artist's style changed. This is the puzzle that scientists face when they try to study the effects of new tools on complex, unstructured things like text, images, or medical records. They need a way to measure the "shape" of the change, not just the size.

This is where the paper by Kevin Christian Wibisono and Yixin Wang comes in. They are statisticians who have invented a new way to ask questions about these messy outcomes. Instead of guessing which part of a text or image changed, they created a method to automatically find the "Maximally Contrasting Feature" (MCF). Think of it as a magical highlighter that scans thousands of documents or images and instantly learns which specific details—like a certain tone of voice, a specific type of blur, or a particular way of organizing thoughts—are the ones that changed the most because of a treatment. They tested this idea with computer simulations using text and images, and the results suggest that their method can successfully spot the exact features that a new tool or intervention alters, even when the researchers didn't know what to look for beforehand.

The Problem: When the Answer Isn't a Number

For a long time, science has been great at measuring simple things. If a doctor wants to know if a new medicine works, they check if the patient's temperature went down by 2 degrees. If a company wants to know if a new website design works, they count if sales went up by 10%. These are "scalar outcomes"—single numbers that are easy to compare.

But the world is full of things that aren't single numbers. Imagine a hospital wants to know if a new AI tool helps doctors write better notes. The "outcome" isn't a number; it's a whole paragraph of text. Or imagine a researcher wants to know if a new camera algorithm makes photos clearer. The outcome is an image. You can't take a photo of a cat and subtract a photo of a dog to get a "difference number." You can't even really say what the "average" note looks like.

Traditionally, scientists tried to solve this by forcing the messy data into a box. They would make a list of rules (a "codebook") to score the text. Maybe they decided to count how many times a doctor used the word "urgent," or how many sentences were in the note. But this is like trying to describe a symphony by only counting the number of violins. You might miss the most important part: the melody, the emotion, or the sudden shift in tempo. If the AI tool changed the tone of the notes but not the word count, the old method would miss it completely.

The Solution: The Magic Highlighter (MCF)

The authors propose a new approach. Instead of guessing what to measure, they let the data teach them what matters. They call this the Maximally Contrasting Feature (MCF).

Imagine you have two piles of notes: one pile written by doctors using the new AI tool, and another pile written by doctors without it. You want to find the "magic highlighter" that, when you run it over the notes, makes the AI pile look as different as possible from the non-AI pile.

The MCF is that highlighter. It's a computer program that learns to assign a score from 0 to 1 to every single note.

If a note looks very much like the "AI style," it gets a score close to 1.
If it looks like the "human style," it gets a score close to 0.

The computer doesn't just guess what "AI style" means. It tries millions of different ways to score the notes, looking for the one way that creates the biggest gap between the two piles. Once it finds the best scoring system, it looks at the notes that got high scores and the notes that got low scores to see what's actually different. Maybe it discovers that the AI notes are always more formal, or maybe they always use a specific template, or perhaps they are just shorter. The method finds the answer without the researcher needing to know the answer in advance.

How It Works: The Detective's Toolkit

The paper explains that this isn't just a guessing game; it's a rigorous mathematical process that accounts for "confounders." In the real world, things are messy. Maybe the doctors who use the AI tool are also the ones who see more complicated patients. If you just compare the notes, you might think the AI made them shorter, when really, the patients just had less to say.

The MCF method fixes this by using a "propensity score." Think of this as a matching game. The computer looks at the background details (like the patient's age, the type of visit, and the doctor's experience) and finds pairs of notes that are almost identical in every way except for whether the AI was used. By comparing these matched pairs, the method isolates the true effect of the AI tool.

The authors also developed a "budgeted" version. Sometimes, the AI changes everything a little bit, or it changes a few things a lot. The budgeted MCF lets researchers say, "Show me the top 10% of notes that changed the most." This helps focus on the most obvious, dramatic changes rather than getting lost in tiny, insignificant shifts.

What They Found: Text, Images, and Surprises

The researchers tested their idea with three different experiments to see if it actually works.

1. The Text Experiments:
They simulated a scenario where a writing tool was supposed to make text more "formal."

The Result: The MCF successfully learned to spot formal language. It gave high scores to sentences that sounded professional and low scores to casual ones.
The Twist: They also tested a trickier scenario where the tool was supposed to make text less formal in some situations (like entertainment) but more formal in others (like family advice). The MCF didn't get confused. It learned a "context-aware" rule: "If it's about music, make it casual; if it's about family, make it serious." It figured out that the "right" answer depends on the situation.
The Safety Test: They also tested if the tool could spot "toxic" language. In one case, the tool made language safer in support groups but more aggressive in heated arguments. The MCF correctly identified that the "safety" feature flipped depending on the context, proving it could handle complex, real-world rules.

2. The Image Experiment:
They used pictures of cells (like those seen under a microscope).

The Result: They simulated a treatment that made the images blurrier. The MCF learned to score the images based on how blurry they were. It successfully separated the sharp images from the blurry ones, even though the number of cells in the pictures stayed the same. It proved that the method works for visual data, not just words.

3. The Double Mystery:
Finally, they looked at a case where both the input and the output were messy. Imagine an AI that suggests a draft, and a human writes the final note. Both the suggestion and the note are text.

The Result: They created a pair of highlighters: one for the AI suggestion and one for the final note. The system learned that when the AI suggestion was "concrete and specific," the final note tended to be "complete and detailed." It found the link between the two messy texts without anyone telling it what to look for.

Why It Matters

This paper suggests a new way to study the world. We are surrounded by unstructured data—emails, videos, medical scans, social media posts. We want to know how new technologies change these things, but we often don't know how to measure the change.

The MCF method offers a way to let the data speak for itself. Instead of forcing a square peg into a round hole by counting words or pixels, it finds the hidden dimensions that actually matter. It suggests that we can discover the most important changes in our complex world, whether it's how an AI changes a doctor's writing or how a new filter changes a photo, without needing to be experts in the specific details beforehand.

The authors are careful to note that these results come from simulations and semi-synthetic experiments. They haven't yet applied this to a real hospital or a live news site, but the simulations show that the math holds up. If this method works in the messy real world, it could help us understand the true impact of the AI tools that are rapidly changing how we live, work, and communicate.

Technical Summary: Causal Inference with Unstructured Outcomes

1. Problem Statement

Traditional causal inference focuses on scalar outcomes (e.g., binary recovery status, earnings, visit counts), where the Average Treatment Effect (ATE) is defined as the difference in mean potential outcomes, $\mathbb{E}[Y(1) - Y(0)]$ . However, modern studies increasingly involve unstructured outcomes such as clinical notes, open-ended survey responses, images, and videos. In these settings, the standard ATE is ill-defined because one cannot meaningfully subtract one text document or image from another.

While a common approach is to pre-specify a feature-scoring function $g$ (e.g., a codebook or topic model) to convert the unstructured object into a scalar, this requires the analyst to know in advance which aspect of the outcome is causally affected. If the relevant change lies outside the pre-specified features (e.g., a subtle shift in tone or texture not captured by a fixed codebook), the analysis may miss the causal signal. The paper addresses the challenge of learning which feature of an unstructured outcome is most causally affected by a treatment without prior specification.

2. Methodology

The Maximally Contrasting Feature (MCF)
The core proposal is the Maximally Contrasting Feature (MCF), a causal query that learns a feature-scoring function $g: \mathcal{Y} \to [0, 1]$ which maximizes the causal contrast between treated and control potential outcomes.

Objective: Find $g^* \in \mathcal{G}$ such that:
$g^* \in \arg\max_{g \in \mathcal{G}} \theta(g), \quad \text{where } \theta(g) = \mathbb{E}[g\{Y(1)\} - g\{Y(0)\}]$
Here, $\mathcal{G}$ is a prespecified class of bounded functions (e.g., neural networks with sigmoid outputs). The function $g$ maps the unstructured outcome to a scalar that exposes the sharpest contrast between the treated and control distributions.

Identification and Estimation

Assumptions: The method relies on standard causal assumptions: SUTVA, Overlap (positivity), and Unconfoundedness (conditional independence of potential outcomes and treatment given covariates $X$ ).
Identification: Under these assumptions, the contrast $\theta(g)$ is identified for any fixed $g$ via inverse propensity weighting (IPW) or outcome regression. The paper proves that if the population criterion has a unique maximizer, the selected feature is identified.
Oracle Interpretation: In the population (oracle) setting, the optimal $g^*$ acts as an indicator function for the region where the density of the treated outcome exceeds that of the control outcome: $g^*(y) = \mathbb{1}\{p_1(y) - p_0(y) > 0\}$ .
Estimation Algorithm:
1. Propensity Score: Estimate the propensity score $\hat{e}(X) = P(A=1|X)$ using a training split.
2. Optimization: On a held-out split, learn the parameters $\beta$ of the scoring function $g(Y; \beta)$ by maximizing the IPW objective:
  $\hat{\theta}_N(\beta) = \frac{1}{N} \sum_{i=1}^N \left( \frac{A_i g(Y_i; \beta)}{\hat{e}(X_i)} - \frac{(1-A_i) g(Y_i; \beta)}{1-\hat{e}(X_i)} \right)$
3. Interpretation: The learned function is interpreted by inspecting outcomes with high vs. low scores (e.g., retrieving nearest neighbors or minimal pairs) to identify recurring patterns.

Extensions

Heterogeneous Effects: The framework extends to covariate-adaptive MCFs, $g(Y, X)$ , which identify features most affected by treatment within specific subpopulations defined by $X$ .
Budgeted MCF: To aid interpretation, a "budgeted" version constrains the function to assign high scores to only a fraction $\tau$ of the population. This prioritizes the most treatment-enriched outcomes (those with the highest "treatment enrichment score" $\Delta(y)/p_{ref}(y)$ ).
Unstructured Treatments: When both treatment ( $A$ ) and outcome ( $Y$ ) are unstructured, the method learns a pair of functions: an influential treatment feature $f(A)$ and a contrasting outcome feature $g(Y)$ . This is estimated by maximizing the conditional covariance between $f(A)$ and the residualized $g(Y)$ , using matched negative-control outcomes to estimate the covariate baseline without nested optimization.

3. Key Contributions

The paper makes five primary contributions:

Proposal of MCF: A new causal query for unstructured outcomes that learns a bounded feature-scoring function capturing the aspect of the outcome with the largest treatment-induced contrast.
Identification and Estimation: Establishment of identification conditions and development of inverse-propensity-weighted (IPW) estimation algorithms with proofs of consistency and asymptotic normality.
Characterization: Theoretical characterization of the population behavior of the MCF, including budgeted and covariate-adaptive versions for better interpretability.
Unstructured Treatments: Extension of the algorithm to settings with unstructured treatments and outcomes by learning paired feature-scoring functions using matched negative-control outcomes.
Empirical Validation: Illustration of the algorithm across text-based outcomes, image-based outcomes, and scenarios with both unstructured treatments and outcomes.

4. Empirical Results

The authors evaluate the MCF algorithm on synthetic and semi-synthetic datasets:

Text Outcomes (Formality & Toxicity):
- In a formality study (GYAFC dataset), the MCF successfully recovered a global formality score in homogeneous settings and adapted to context-dependent directions (e.g., increasing formality in family contexts but decreasing it in entertainment contexts) in heterogeneous settings.
- In a toxicity study (ParaDetox dataset), the algorithm recovered global non-toxicity and correctly identified context-dependent shifts where treatment increased toxicity in high-conflict settings but decreased it in support settings.
- The method outperformed pre-specified topic models (LDA), which failed to isolate the specific causal dimension (formality) when the contrast was distributed across topics.
- The MCF also successfully isolated specific combinations of attributes (e.g., formality + punctuation) when treatment induced simultaneous changes.
Image Outcomes:
- Using synthetic cell images (BBBC005v1), the MCF isolated the "blur" style dimension induced by treatment while holding cell count (content) constant. The learned score effectively separated low-blur from high-blur images.
Unstructured Treatments and Outcomes:
- In a text-to-text setting (AI suggestions as treatment, final notes as outcome), the paired MIF-MCF algorithm successfully identified linked directions (e.g., concrete suggestions leading to complete plans) after adjusting for covariates.

5. Significance and Claims

The paper claims that the MCF provides a data-driven, exploratory tool for causal inference with unstructured outcomes. Its significance lies in:

Avoiding Pre-specification Bias: It removes the need for analysts to manually define codebooks or features before analysis, allowing the data to reveal which aspects of the outcome are causally shifted.
Interpretability: By learning a scoring function and then inspecting high/low-scoring examples, the method bridges the gap between high-dimensional embeddings and human-interpretable causal mechanisms.
Handling Complexity: It addresses the "ill-defined" nature of ATEs for unstructured objects by reframing the causal question around the contrast in a learned feature space rather than a direct subtraction of objects.
Flexibility: The framework accommodates heterogeneous effects and bidirectional unstructured data (treatment and outcome), expanding the scope of causal questions that can be asked in fields like healthcare, public policy, and social science.

The authors maintain a modest tone, positioning the MCF as an exploratory map that identifies where treatment-induced contrasts are concentrated, rather than a confirmatory test for a pre-named attribute. The labels (e.g., "formality," "blur") are interpretations derived from the learned scores and subsequent domain expert inspection, not inherent properties of the algorithm itself.