What is the Causal Effect of a Conversation? Estimands and Inference in AI Mediated Conversations

Technical Summary: Estimands and Inference in AI-Mediated Conversations

Problem Statement
Political scientists increasingly utilize conversations as causal treatments, ranging from human canvassers to generative artificial intelligence (AI). While the interactive, responsive nature of conversations offers high ecological validity and scalability (particularly with AI), it introduces fundamental challenges for causal inference. Unlike static interventions (e.g., leaflets or fixed text), conversations are jointly produced by the respondent and the interlocutor. Consequently, the realized conversation is endogenous to the respondent's characteristics (e.g., hostility, curiosity, or resistance).

The core problem is that random assignment to a conversational condition (e.g., a "civil" vs. "uncivil" AI policy) identifies the effect of the assignment, but does not necessarily identify the effect of the realized conversation or specific features within it (e.g., the level of incivility). Because the conversation unfolds dynamically based on the respondent's inputs, variation in the realized treatment is systematically related to respondent traits that also affect outcomes. This creates confounding that standard randomized designs cannot resolve without additional assumptions. The paper argues that treating the "conversation" as a monolithic treatment obscures distinct theoretical objects (assignment, policy, message, feature) that require different causal definitions and identification strategies.

Methodology and Framework
The authors develop a potential outcomes framework adapted for sequential, interactive processes. They formalize the conversation as a path-dependent process involving:

• Assignment ($Z_i$): The randomized condition (e.g., prompt or policy).
• Policy ($\pi$): The rule governing the interlocutor's responses to history.
• History ($H_{it}$): The sequence of prior messages.
• Realized Conversation ($C_i$): The full path of messages generated by the interaction of policy, history, and respondent behavior ($R_i$).
• Features ($D_i, B_i$): Measured and unmeasured latent features of the conversation (e.g., tone, civility).

The framework distinguishes between the design-induced intervention (what is randomized) and the realized exposure (what the respondent actually experiences). It utilizes Directed Acyclic Graphs (DAGs) to illustrate how respondent traits ($U_i$) influence both the conversational path and the outcome, creating endogeneity even under random assignment.

The paper defines a taxonomy of causal estimands, each corresponding to a different theoretical object and requiring distinct identifying assumptions:

1. Assignment Effect ($\tau_Z$): The Intent-to-Treat (ITT) effect of being assigned to a condition. Identified under standard randomization assumptions (Consistency, No Interference, Ignorability, Positivity).
2. Opening Message Effect ($\tau_{A0}$): The effect of the initial message. Identifiable if the message is directly randomized, but often confounded if the message is a proxy for a latent feature.
3. Policy Effect ($\tau_\pi$): The effect of deploying a specific conversational regime. Identifiable under randomization of the policy, assuming stable implementation. However, this is an aggregate effect that averages over diverse conversational trajectories.
4. Message Effect ($\tau_{A_t}$): The effect of a specific message conditional on history. Requires Sequential Ignorability (the next message is as-if random conditional on history). The paper argues this is often violated because history is endogenous to respondent traits.
5. Feature Effect ($\tau_D$): The effect of a latent feature (e.g., incivility). Requires that variation in the feature be decoupled from other unmeasured message components (bundling bias).
6. Realized Conversation Effect ($\tau_C$): The effect of the full path. Requires Conversation Ignorability, which is generally untestable and unlikely to hold as conversations are jointly generated.
7. Dosage Effect ($\tau_E$): The effect of cumulative exposure to a feature. Inherits confounding from the feature-level estimand.

Key Results and Propositions
The paper presents several propositions formalizing the limits of identification in conversational settings:

• Proposition 1 (Selection Bias): Comparing outcomes across different messages fails to isolate the message effect because the messages are generated by different histories, which are driven by unobserved respondent traits ($U_i$).
• Proposition 2 (Feature Bundling Bias): Variation in a feature of interest (e.g., tone) is typically bundled with other unmeasured aspects of the message. Randomizing the message does not isolate the feature effect.
• Proposition 3 (Endogeneity of Paths): The realized conversation is both a treatment and an indicator of respondent characteristics. Comparing realized paths confounds the treatment effect with respondent heterogeneity.
• Proposition 4 (Dosage Inheritance): Aggregating features into a dosage measure does not resolve the underlying confounding present at the feature level.

Implications for Research Design
The authors propose several design strategies to align estimands with research goals:

• Match Estimand to Design: Researchers must explicitly define whether they are interested in the assignment, policy, or specific features, and design the randomization accordingly.
• Turn-Level Randomization: Instead of randomizing the entire condition, researchers can randomize specific messages at specific turns (conditional on history). This creates a local Average Treatment Effect (LATE) for specific conversational states, reducing endogeneity.
• Conditional Average Treatment Effects (CATE): By conditioning on representations of history (e.g., "civil" vs. "hostile" context) rather than the full high-dimensional history, researchers can estimate how message effects vary by context.
• Average Marginal Component Effect (AMCE): Aggregating CATEs across the observed distribution of conversational contexts to summarize average message effects.
• Instrumental Variables (LATE): When realized exposure cannot be directly manipulated, assignment can serve as an instrument for the realized feature, identifying a Local Average Treatment Effect for "compliers" (those whose conversation was shifted by the assignment).
• Constrained Protocols: Restricting the message space (e.g., multiple-choice responses) to improve empirical support for comparisons, though this may trade off with ecological validity.

Significance and Claims
The paper claims to provide a common language for defining and interpreting causal effects in conversational settings, particularly those mediated by AI. Its primary contribution is not to suggest that researchers should retreat to only easily identified estimands (like ITT), but to clarify the trade-offs involved in studying complex, theoretically rich objects like "persuasion" or "deliberation."

The authors argue that the gap between what is assigned and what is experienced is a fundamental feature of social interaction, not merely a methodological failure. By making the assumptions required to move from assignment to realized features explicit, the framework allows scholars to:

1. Better align empirical strategies with theoretical goals.
2. Interpret findings with appropriate caution regarding confounding.
3. Innovate in research design (e.g., sequential randomization) to target specific causal objects.

The paper concludes that while conversations are powerful because they are adaptive and co-produced, these same features make causal effects harder to define and identify. The framework serves to navigate this complexity rather than eliminate it.