Technical Summary: Leveraging Large Language Models for Causal Discovery

Problem Statement

Causal discovery aims to reconstruct causal graphs from observational data to enable prediction and intervention analysis. While statistical methods (constraint-based and score-based) exist, they rely heavily on assumptions (e.g., faithfulness, causal sufficiency) and sample size, often yielding equivalence classes rather than unique Directed Acyclic Graphs (DAGs). Incorporating domain expert knowledge can refine these results, but manually encoding constraints is labor-intensive and often treats expert input as "ground truth," offering limited mechanisms to resolve conflicts between expert intuition and statistical evidence.

Recent approaches utilize Large Language Models (LLMs) to extract causal constraints from variable metadata. However, LLMs are imperfect experts prone to hallucination and memorization of public benchmarks. A critical challenge is distinguishing genuine causal reasoning from the verbatim retrieval of memorized graphs, particularly when standard benchmarks (e.g., Asia, Sachs) are likely present in LLM pre-training corpora.

Methodology

The paper proposes ABAPC-LLM, a hybrid framework that integrates LLM-derived semantic constraints with statistical evidence using Causal Assumption-Based Argumentation (Causal ABA).

1. The Argumentation Framework (Causal ABA)

Instead of treating constraints as hard rules, the framework models causal discovery as an argumentation problem:

• Assumptions: Candidate causal edges and conditional independence (CI) statements are treated as defeasible assumptions.
• Rules: Logical rules encode acyclicity and d-separation principles.
• Attacks: Incompatible constraints (e.g., a CI statement implying independence vs. a constraint implying a direct edge) attack each other.
• Resolution: The system computes "stable extensions," retaining only the largest subset of assumptions that are jointly defensible. This allows the framework to transparently expose and resolve conflicts between data and expert (or LLM) knowledge, rather than blindly enforcing one source.

2. LLM Integration Pipeline

The method treats LLMs as imperfect experts to elicit structural constraints:

• Elicitation: An LLM is prompted with variable names and descriptions to generate lists of Required Directions (edges that must exist) and Forbidden Directions (edges that cannot exist).
• Consensus Filtering: To mitigate stochasticity and hallucination, the LLM is queried five times. Only constraints appearing in the intersection of all five runs (unanimous consensus) are retained. This conservative approach prioritizes high precision over recall.
• Integration: These consensus constraints are encoded as facts within the Causal ABA framework. Crucially, they are used to orient existing edges identified by statistical tests, not to introduce new edges. If a statistical test removes an edge (skeleton reduction), the LLM cannot force it back.

3. Evaluation Protocol: CauseNet Synthetic Benchmarks

To address the risk of LLM memorization, the authors introduce a novel evaluation protocol using CauseNet, a large-scale knowledge graph of cause-and-effect relationships:

1. Structural Scaffolding: Random DAGs are generated using Erdős-Rényi, Scale-Free, or Lower-Triangle topologies.
2. Semantic Grounding: The random DAG structure is mapped to a sub-graph within CauseNet via induced sub-graph isomorphism.
3. Heuristic Selection: Candidate mappings are ranked by semantic compactness, node specificity, and structural-semantic correlation to ensure the resulting graph is semantically coherent yet structurally novel.
4. Data Generation: Observational data is sampled from the grounded DAGs. This ensures the LLM is tested on novel semantic combinations rather than memorized benchmark graphs.

Key Contributions

1. ABAPC-LLM Framework: A principled, conservative approach to hybrid causal discovery that uses argumentation to integrate LLM-derived constraints with data-driven CI evidence. It treats LLM output as defeasible, allowing statistical evidence to overrule semantic constraints when conflicts arise.
2. Consensus-Based Elicitation: A pipeline that uses multi-run unanimity filtering to extract high-precision directional constraints from LLMs, reducing the risk of incorporating spurious causal links.
3. Mitigation of Memorization Bias: The introduction of the CauseNet-based synthetic benchmark protocol, which grounds random DAGs in real-world concepts to evaluate LLM generalization beyond memorized public datasets.
4. Empirical Validation: Comprehensive evaluation showing competitive performance on standard benchmarks and superior performance on the novel synthetic benchmarks, particularly in structural recovery metrics (SHD, F1-score).

Experimental Results

• Synthetic Benchmarks (CauseNet): ABAPC-LLM achieved the lowest Structural Hamming Distance (SHD) and highest recall across all node counts (5, 10, 15). It consistently outperformed standard baselines (MPC, FGS, NOTEARS) and the argumentation-only variant (ABAPC), demonstrating that semantic constraints improve structural accuracy when conflicts are resolved via argumentation.
• Standard Benchmarks (bnlearn): The method performed competitively on standard datasets (e.g., sachs, child, asia). In some cases (e.g., earthquake), standard methods performed well, but ABAPC-LLM maintained strong performance, particularly on larger, more complex graphs.
• Constraint Quality Analysis: The study revealed a "synergy" effect: when LLM constraints were high-quality, they significantly boosted final graph accuracy. However, low-quality LLM constraints could degrade performance if they forced the relaxation of strong statistical evidence, highlighting the importance of the argumentation layer in filtering noise.
• Runtime: The approach adds overhead for solving the argumentation framework but remains tractable for graphs up to ~20 variables, with runtimes in seconds for smaller datasets.

Significance and Claims

The paper claims that ABAPC-LLM offers a conservative and auditable method for leveraging LLMs in causal discovery. By treating LLM knowledge as defeasible assumptions rather than ground truth, the framework avoids the pitfalls of hallucination and memorization while capitalizing on the semantic signal LLMs provide.

The authors emphasize that their approach does not claim LLMs are perfect causal reasoners. Instead, it positions LLMs as a source of structured, semantic priors that can be rigorously tested against data. The use of the CauseNet protocol is presented as a necessary step to evaluate generalization, arguing that high performance on standard benchmarks may simply reflect data contamination. The work advocates for a future where LLMs serve as "auditable sources of causal directions," mediated by formal consistency checks, rather than autonomous decision-makers.