🤖 machine learning

Dynamic Expert-Guided Model Averaging for Causal Discovery

This paper proposes a dynamic expert-guided model averaging framework that selectively queries imperfect experts to resolve uncertainties among diverse causal discovery algorithms, thereby improving performance on both clean and noisy data by distinguishing between edge existence and orientation.

Original authors: Adrick Tench, Thomas Demeester

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adrick Tench, Thomas Demeester

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out the secret recipe for a complex dish, like a perfect stew. You have a team of ten different chefs (algorithms). Each chef tastes the ingredients and draws a map of how they think the flavors connect.

Chef A thinks salt causes the meat to be tender.
Chef B thinks the meat causes the saltiness.
Chef C isn't sure if they are connected at all.
Chef D thinks they are connected, but the direction is wrong.

In the real world, these chefs often disagree, especially if the ingredients (data) are messy or noisy. If you just pick one chef, you might get a bad recipe. If you just take a "majority vote," you might still get a confused map because the chefs are all looking at the same confusing data.

This paper introduces a new way to combine these chefs into a super-team, using a dynamic expert to help them agree.

The Core Problem: Too Many Chefs, Not Enough Clarity

The authors note that in the field of "causal discovery" (figuring out what causes what), there are dozens of different computer methods. None of them is perfect. Sometimes the data is clean, but often it's noisy, like a kitchen with a loud radio playing in the background. Because no single method works best in every situation, the authors suggest we should use all of them together (an "ensemble").

However, simply averaging their answers isn't enough. If the chefs are all confused about the same thing, the group answer is still wrong.

The Solution: The "Dynamic Expert"

The authors propose a system where the computer team doesn't just vote; it asks for help from an expert (which could be a human specialist or an AI like a Large Language Model) only when it's truly stuck.

Here is how their system works, broken down into two simple steps:

1. The "Is it there?" Question (Edge Existence)

First, the team decides: Is there a connection between these two ingredients at all?

The Rule: If 80% of the chefs say "Yes, they are connected," the system adds the connection.
The Gray Area: If the chefs are split (say, 40% say "Yes" and 60% say "No"), the system doesn't guess. Instead, it asks the Expert: "Is there a real link here?"
The Catch: The Expert has a limited budget. You can't ask them every single question because they are expensive (or slow). So, the system only asks when the computer chefs are really arguing.

2. The "Which way?" Question (Edge Orientation)

Once the system agrees a connection exists, it needs to know the direction: Does Salt cause Meat, or does Meat cause Salt?

The Rule: If 90% of the chefs agree on the direction, the system picks that direction.
The Gray Area: If the chefs are split on the direction (some say A→B, others say B→A), the system asks the Expert again: "Which way does the causality flow?"

Why This is Smart (The Analogy)

Think of the computer algorithms as a group of students taking a test.

Standard Voting: You just take the most common answer. If the students are all guessing on a hard question, the majority might still be wrong.
This Paper's Method: The teacher (the system) looks at the test. If 90% of students agree on an answer, the teacher marks it correct without checking. But if the class is split 50/50 on a question, the teacher stops and asks a Tutor (the expert) for the answer.
The Benefit: The teacher saves time and money by only asking the Tutor when the students are truly confused. This ensures the final answer is better than just the students voting alone, and better than just asking the Tutor for everything (which would be too slow or expensive).

What They Found

The authors tested this method on various "kitchens" (datasets) ranging from simple medical scenarios to complex sports outcomes. They compared their method against:

Just using one single chef.
Just voting among the chefs without help.
Asking the expert for help on every question (which is inefficient).

The Result: Their "Dynamic Expert" method consistently produced better recipes (more accurate causal graphs) than the other methods, even when the ingredients were messy (noisy data).

Important Limitations (What the Paper Does Not Claim)

It's not magic: The "Expert" isn't perfect. In their experiments, they used a "simulated" expert who got things right about 70-90% of the time. Even with an imperfect expert, the system worked better than without one.
It's not a crystal ball: The paper admits it cannot guarantee a perfect result in every theoretical scenario. It relies on the expert being slightly better than random guessing.
It's about the method, not the medicine: While they used medical datasets (like diagnosing diseases), the paper focuses entirely on the mathematical method of combining algorithms. It does not claim this specific tool is ready to diagnose patients in a hospital tomorrow, nor does it discuss future clinical applications.

Summary

This paper presents a smart way to combine many different computer programs to figure out cause-and-effect relationships. Instead of letting the computers guess or asking an expert for every single detail, the system acts like a smart manager: it lets the computers vote when they agree, and only calls in the expert when the computers are truly confused. This saves resources and leads to more accurate results.

Technical Summary: Dynamic Expert-Guided Model-Averaging for Causal Discovery

Problem Statement

Practitioners of causal discovery face a "dizzying array" of algorithms without a clear consensus on which method performs best for specific use cases. This diversity, combined with the realities of limited and noisy data, often forces reliance on expert knowledge, as individual algorithms frequently violate theoretical assumptions (e.g., causal sufficiency) or fail to consistently outperform others on real-world data. While ensembling (model averaging) is a natural strategy to improve robustness, existing methods for causal discovery ensembling are limited. Some rely on unrealistic metadata about ground-truth graphs, while others focus on static priors rather than dynamically resolving discrepancies. Furthermore, the integration of expert knowledge has typically been limited to static constraints or specific algorithms, rather than serving as a mediator for a diverse ensemble.

Methodology

The authors propose a Dynamic Expert-Guided Model-Averaging strategy that integrates a diverse set of causal discovery algorithms with an imperfect expert (human or Large Language Model). The core innovation lies in decoupling the determination of edge existence from edge orientation and using the expert selectively to resolve disagreements.

Key Methodological Components:

Separation of Existence and Orientation:
- Edge Existence: The method relies primarily on data-driven algorithms to detect the presence of edges. It acknowledges that while algorithms may disagree on orientation, they often agree on the existence of a connection.
- Edge Orientation: Once an edge is established, the method determines its direction. It recognizes that algorithms often struggle with orientation (producing CPDAGs with undirected edges) and that experts (particularly LLMs) may be better at determining causal ordering than detecting the existence of a relationship.
Dynamic Querying Strategy:
- The method runs a set of $M$ causal discovery algorithms to produce a set of graphs $\mathcal{G}$ .
- Edge Addition: Edges are considered for addition based on a frequency threshold $\theta_1$ . If the percentage of models containing a direct edge exceeds $\theta_1$ but falls below a majority vote (0.5), the system queries the expert to validate the edge's existence.
- Orientation Determination: For accepted edges, if multiple valid orientations exist, the method checks if a specific orientation exceeds a threshold $\theta_2$ among models containing the edge. If not, or if the orientation is ambiguous, the expert is queried to determine the direction.
- Query Budget ( $B$ ): To reflect realistic constraints on expert access (time/cost), the method operates under a fixed query budget. Once the budget is exhausted, decisions default to majority voting.
- Disagreement-Driven Queries: The system prioritizes querying the expert only for edges where the ensemble exhibits significant disagreement, maximizing the value of limited expert input.
Expert Model:
- The method treats the expert as "imperfect," capable of errors. In experiments, a simulated expert provides correct answers with probabilities $p_{exist}^e$ and $p_{orient}^e$ .
- The approach assumes the expert answers based on plausible causal ordering (direct or indirect chains) rather than strictly direct edges, reflecting the difficulty of eliciting precise structural knowledge in practice.

Key Contributions

The paper makes three primary contributions:

Novel Ensembling Strategy: It introduces the first method to use dynamically requested expert knowledge specifically for mediating discrepancies in causal discovery model averaging.
Decoupled Querying: By treating edge existence and orientation separately, the method leverages the complementary strengths of data-driven algorithms (detection) and expert input (direction), selectively querying the expert only in cases of high uncertainty or disagreement.
Empirical Validation: The method is evaluated against strong baselines (including existing ensembling methods and individual algorithms) on well-known real-world graphs using both clean and noisy data, demonstrating consistent performance improvements.

Experimental Results

The authors evaluated their method on six benchmark networks (e.g., Asia, Sports, Alarm, Pathfinder) with varying sample sizes ( $n \in \{100, 1000, 10000, 100000\}$ ) and noise levels.

Performance: The proposed method consistently outperformed strong baselines, including individual algorithms (PC-Stable, FCI, Tabu, GES, etc.) and existing ensembling approaches (Bayesys Model Avg, Aslani & Mohebbi).
Metrics: Improvements were observed across Balanced Scoring Function (BSF), F1-score, Structural Hamming Distance (SHD), and Structural Intervention Distance (SID). The method showed broad improvements in recall while maintaining comparable precision to other ensembling methods.
Comparison to Tabu-AL: The method significantly outperformed Tabu-AL (a baseline using dynamic expert knowledge on a single algorithm) by at least 11% on average across most metrics, highlighting the value of ensembling diverse algorithms with expert mediation.
LLM Case Study: Experiments using an LLM (gpt-5-nano) as the expert showed performance consistent with the simulated expert parameters ( $p_{exist}^e \approx 70\%$ , $p_{orient}^e \approx 90\%$ ), validating the feasibility of using LLMs in this framework.
Query Budget: A moderate budget of $2N$ (where $N$ is the number of nodes) provided the best balance between precision and recall.

Significance and Claims

The paper positions this work as the first method to leverage expert knowledge for the ensembling of causal discovery algorithms. Its significance is framed around the practical necessity of handling real-world data constraints where no single algorithm is universally superior.

Modest Claims: The authors explicitly state that their method provides no formal guarantees of recovering the true causal graph under ideal settings. Instead, it relies on empirical validation to combine the strengths of diverse algorithms with imperfect experts under plausible real-world conditions.
Practicality: The approach is designed for scenarios where access to experts is limited and imperfect, offering a robust alternative to relying on a single algorithm or static priors.
Future Work: The authors note limitations, including the use of synthetic categorical data and the lack of formal theoretical guarantees. They suggest future work could explore different data types, noise distributions, and adaptive weighting of models based on predicted performance.

In summary, the paper presents a pragmatic, flexible framework that uses targeted expert intervention to resolve uncertainties in causal ensembles, demonstrating that such a strategy yields superior structural accuracy compared to existing methods on both clean and noisy data.