Evaluating Bivariate Causal Statements Based on Mutual Compatibility

Technical Summary: Evaluating Bivariate Causal Statements Based on Mutual Compatibility

Problem Statement
In many real-world domains such as medicine and economics, establishing causal ground truth through experimentation is often infeasible or prohibitively expensive. Consequently, causal inference relies heavily on expert knowledge or, increasingly, on outputs from Large Language Models (LLMs). These sources typically provide bivariate causal statements (pairwise cause-effect relationships) rather than complete multivariate causal graphs. While humans and LLMs find it easier to assess pairwise relations, aggregating these statements into a coherent global model is challenging. Existing methods for evaluating causal claims without ground truth often rely on consistency with data or specific discovery algorithms. This paper addresses the fundamental question of how to assess the reliability of collections of bivariate causal statements when no ground truth is available, specifically by evaluating their mutual compatibility.

Methodology
The authors propose two distinct frameworks for evaluating bivariate causal statements, depending on the nature of the information provided: quantitative (linear coefficients) or qualitative (graphical existence/direction).

1. Compatibility for Linear Bivariate Causal Statements
This approach targets statements quantifying total pairwise causal effects via linear coefficients ($\alpha_{ij}$).

• Theoretical Foundation: The authors demonstrate that for any acyclic list of linear bivariate statements, there exists a unique multivariate linear Structural Equation Model (SEM) whose pairwise marginals exactly match the given statements. Because a unique model always exists, traditional "hard" compatibility constraints (beyond acyclicity) do not apply.
• The Confounding Postulate: To evaluate plausibility, the authors introduce a postulate: a generic multivariate causal model should not exhibit more confounding than its pairwise marginal models. The intuition is that marginalizing a multivariate model reduces observed causal pathways, thereby increasing unobserved confounding in the marginals. If the induced multivariate model requires less confounding than the marginals to explain the data, it implies a specific, unlikely cancellation of causal pathways (violating the postulate).
• Scoring Mechanism:
  • Confounding Measure: They define a confounding score based on the squared residual of covariances not explained by observed causal paths. For a bivariate model, this is $(\Sigma_{ij} - \alpha_{ij}\Sigma_{ii})^2$. For the multivariate model, it is the squared residual after accounting for all directed paths and fully observed back-door paths.
  • Compatibility Score: The score is defined as the sum over all pairs of the difference between the bivariate confounding and the multivariate confounding:
    $$\text{comp}(\Sigma, A) = \sum_{i<j} (C^{\text{biv}}_{ij} - C^{\text{mult}}_{ij})$$
  • Interpretation: A negative score indicates a violation of the confounding postulate, serving as evidence to falsify the statements. A positive score does not guarantee correctness but indicates the statements induce a plausible model.
• Theoretical Guarantees: The paper proves that for random causal models satisfying unbiasedness and independence of mechanisms, the expected compatibility score is positive. It also provides finite-sample concentration bounds showing that empirical estimates of the score converge to the true score with polynomial sample complexity.

2. Incompatibility for Graphical Bivariate Causal Statements
This approach targets qualitative statements indicating the presence/absence of causal effects and confounding, modeled as Acyclic Directed Mixed Graphs (ADMGs).

• Graphical Compatibility: Under the faithfulness assumption, a collection of bivariate ADMGs is compatible if and only if they are the marginalizations of a single global ADMG. This requires three conditions:
  1. The directed part of the union graph is acyclic.
  2. The directed part is transitively closed.
  3. If a confounding path exists between two nodes, a bidirected edge must exist between them.
• Incompatibility Score: Since finding the closest compatible graph is NP-hard (related to the Acyclic Transitivity Editing problem), the authors propose a heuristic algorithm to approximate the minimum number of edge edits (additions/deletions) required to satisfy the three conditions. The score is the sum of edits needed to:
  1. Break cycles (Feedback Arc Set).
  2. Enforce transitive closure.
  3. Enforce the confounding path closure property.
• Properties: The heuristic score is an upper bound on the true incompatibility. If the score is zero, the statements are perfectly compatible; if large, it indicates significant inconsistency.

Key Results

• Synthetic Experiments:
  • Linear Case: The compatibility score successfully distinguishes correct from incorrect statements. As noise (error) is added to ground truth coefficients, the fraction of statements with positive scores decreases. The score is robust to hidden variables and improves with the number of observed variables and model density.
  • Graphical Case: The incompatibility score increases monotonically with the number of injected errors in synthetic graphs. For sparse graphs, it closely approximates the true error count; for dense graphs, it serves as a reliable upper bound.
• LLM Evaluation:
  • The authors applied these methods to causal claims generated by various LLMs (e.g., Claude, GPT, Gemma, Qwen) regarding socioeconomic indicators (population density, literacy, income, etc.).
  • Linear Results: Compatibility scores varied significantly across models, with higher-capacity models generally achieving higher (more plausible) scores. Many LLMs received negative scores, indicating their generated statements were internally inconsistent.
  • Graphical Results: Most LLM outputs resulted in non-zero incompatibility scores. Interestingly, some low-scoring models produced extremely dense graphs (claiming confounding everywhere), which trivially satisfied compatibility conditions but lacked informativeness. When filtering for informative graphs (lower edge density), a correlation emerged between lower incompatibility scores and higher model capacity.

Significance and Claims
The paper claims to provide a foundational method for assessing the reliability of causal information derived from human experts or AI in the absence of ground truth.

• Falsification vs. Verification: The authors emphasize that their methods are primarily tools for falsification. A negative compatibility score or a high incompatibility score provides evidence that a set of statements is incorrect or inconsistent. A positive score does not prove correctness but indicates the statements pass a "sanity check" regarding mutual plausibility.
• Independence from Faithfulness (Linear Case): A key contribution is the linear framework's ability to evaluate statements without relying on the faithfulness assumption, which is often violated in practice.
• Practical Application: The work demonstrates that these metrics can differentiate between the quality of causal reasoning in different LLMs, suggesting utility for post-training filtering, validation pipelines, or automated evaluation of AI-generated causal claims.

The authors remain modest, noting that high scores do not guarantee truth and that future work could extend these frameworks to incomplete lists of statements or incorporate statements over small subsets of variables.