Technical Summary: Causal Ensemble Agent (CEA)

Problem Statement

Causal discovery aims to uncover causal structures from observational data, a task critical for decision-making in domains like healthcare and economics. However, identifying accurate causal graphs remains challenging due to two primary limitations in existing approaches:

1. Divergence in Statistical Methods: Different Statistical Causal Discovery (SCD) algorithms often produce conflicting results on the same dataset due to distinct inductive biases and objective functions. While ensemble methods exist, they typically rely on numerical criteria or manually designed weighting schemes, failing to leverage rich domain-specific information such as feature descriptions.
2. Limitations of LLM-based Discovery: Recent approaches using Large Language Models (LLMs) to directly infer causal relations via queries often suffer from unreliability and hallucination. These methods can be misled by vague variable names or incomplete descriptions and frequently lack alignment with the actual data, making them prone to generating incorrect causal assertions.

Methodology: Causal Ensemble Agent (CEA)

The authors propose the Causal Ensemble Agent (CEA), a hierarchical ensemble framework that treats the LLM not as a direct causal expert, but as a meta-referee. CEA aggregates structural insights from multiple heterogeneous SCD experts and dynamically reweights them only when necessary.

Core Framework

The framework operates in a coarse-to-fine manner across three hierarchical layers:

1. Graph Skeletons: Identifying undirected adjacencies.
2. V-structures: Identifying unshielded colliders ($A \to B \leftarrow C$).
3. Edge Orientations: Determining the direction of edges.

Key Mechanisms

• Linear Opinion Pooling: For each candidate relation, CEA computes an empirical support score ($\hat{p}_i(r)$) for each expert $e_i$ using bootstrap resampling. These scores are aggregated via linear opinion pooling: $\hat{p}(r) = \sum \alpha_i \hat{p}_i(r)$, initially with uniform weights ($\alpha_i = 1/m$).
• Margin-Aware Dispute Detection: The system identifies "disputed" relations where the aggregated confidence is close to the decision boundary (i.e., $|\hat{p}(r) - \tau_\kappa| \le \delta \tau_\kappa$). Relations far from the boundary are accepted without LLM intervention.
• LLM as Meta-Referee: For disputed relations, an LLM is invoked to perform a meta-analysis. It does not predict the relation directly. Instead, it evaluates:
  • Meta Context: Variable semantics, dataset profiles (statistics, sample-to-feature ratios), and the type of structural relation (skeleton vs. orientation).
  • Expert Opinions: The methodology, assumptions, and specific stance (support/oppose) of each expert.
  • Output: The LLM assigns a raw weight score ($w_i \in [0, 10]$) to each expert for that specific relation. These scores are normalized to update the pooling weights ($\alpha'_i$), refining the support score to $\hat{p}'(r)$.
• Hierarchical Construction: The final Directed Acyclic Graph (DAG) is constructed by integrating the consensus skeleton, valid v-structures, and resolved edge orientations, ensuring acyclicity through Meek-rule closure and cycle checks.

Key Contributions

1. Redefining the LLM Role: The paper introduces a novel role for LLMs in causal discovery as meta-referees that adjudicate among competing, data-driven hypotheses. This allows for the integration of domain knowledge without sacrificing data alignment or introducing hallucinated edges.
2. Hierarchical Ensemble Framework: CEA aggregates expert beliefs in a coarse-to-fine manner (skeletons, v-structures, orientations) with margin-aware, LLM-guided expert reweighting. This ensures the LLM is only invoked for uncertain cases, reducing computational cost and hallucination risks.
3. Empirical Superiority: Extensive experiments demonstrate that CEA achieves the strongest overall performance across diverse synthetic and real-world benchmarks, outperforming both individual SCD methods and existing ensemble or LLM-based baselines.

Experimental Results

The authors evaluated CEA on eight datasets (five synthetic: Alarm, Asia, Child, Insurance, Sachs; three real-world: Alzheimer, Arctic Sea, Sangiovese) using two metrics: RNB (Ratio between Normalized Hamming Distance and Baseline Hamming Distance, lower is better) and AUC (Area Under the Precision-Recall Curve, higher is better).

• Performance: CEA achieved the best results across all datasets, showing an average relative RNB reduction of 28.3% and an AUC improvement of 6.2% compared to the strongest non-CEA baselines.
• Robustness: CEA demonstrated consistent performance across different sample sizes and domains, effectively mitigating the "solver-selection" problem where individual methods fail in specific domains.
• Efficiency: By restricting LLM queries to disputed relations, CEA reduced LLM query costs by 84.0% to 88.8% compared to exhaustive pairwise LLM querying.
• Ablation Studies:
  • Removing LLM-guided reweighting (using uniform weights) caused a significant performance drop (41.9% RNB increase), confirming the necessity of context-aware calibration.
  • Using the LLM as a direct causal expert (instead of a meta-referee) also degraded performance and raised concerns about benchmark leakage (memorization of graph structures).
  • The LLM's ability to assign weights aligned with empirical expert performance (e.g., down-weighting LiNGAM on discrete data) was verified through weight-performance alignment analysis.

Significance and Claims

The paper claims that CEA effectively reconciles the conflicting outputs of heterogeneous statistical methods by leveraging LLMs for context-aware calibration rather than direct prediction. By grounding the final graph in data-driven SCD outputs and using the LLM only to resolve uncertainties based on domain knowledge and dataset characteristics, CEA offers a robust, scalable, and reliable approach to causal discovery. The authors emphasize that this approach mitigates the risks of hallucination and benchmark leakage associated with direct LLM querying while significantly improving cross-domain robustness.