📊 statistics

Causal Preference Elicitation

This paper introduces causal preference elicitation, a Bayesian framework that actively queries experts on local edge relations to efficiently concentrate posterior distributions over directed acyclic graphs and improve causal discovery accuracy under limited query budgets.

Original authors: Edwin V. Bonilla, He Zhao, Daniel M. Steinberg

Published 2026-06-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Edwin V. Bonilla, He Zhao, Daniel M. Steinberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Blindfolded Architect"

Imagine you are an architect trying to draw a map of a complex city’s underground subway system. You have never seen the tunnels, but you have a pile of train schedules (observational data). By looking at when trains arrive and leave, you can guess which stations are connected.

However, there’s a big problem:

Ambiguity: Just because Train A leaves Station X and Train B arrives at Station Y doesn’t mean there is a direct tunnel between them. There might be a hidden transfer station.
Direction: Even if you know they are connected, you don’t know if the tunnel goes X → Y or Y → X.
Noise: The schedules are messy. Sometimes trains are late, or data is missing.

In science (like biology or medicine), this is called Causal Discovery. We try to figure out "what causes what" (the subway map) using only observational data. But often, the data is too noisy to give us a clear picture.

The Old Way: The "Static Consultant"

Traditionally, scientists would ask an expert (a senior engineer who knows the city) for help. But they usually did it in a rigid way:

The "Hard Constraint" Approach: The scientist says, "Please tell me everything you know about the tunnels before I start drawing." The expert writes down a list of rules. The scientist then forces their map to obey those rules.
The Problem: This assumes the expert is 100% correct and remembers everything perfectly. It also wastes the expert’s time. If the expert is unsure about a specific tunnel, the scientist still treats their guess as a hard fact. Plus, the scientist doesn’t ask the right questions; they just dump all their uncertainty on the expert at once.

The New Solution: CaPE (The "Active Interviewer")

The authors propose a new method called CaPE (Causal Preference Elicitation). Think of CaPE not as a static consultant, but as a smart, active interviewer.

Instead of asking the expert to draw the whole map at once, CaPE plays a game of "20 Questions" with the expert, but with a twist: it uses math to decide which question to ask next.

How It Works (The Analogy)

The Starting Point (The Rough Sketch):
CaPE starts with the "blindfolded architect’s" rough guess based on the train schedules. This guess is fuzzy. It says, "I’m 60% sure there’s a tunnel between X and Y, and 40% sure it goes the other way." This is called the Posterior Distribution.
The Expert Model (The Noisy Oracle):
CaPE acknowledges that experts are human. They might be tired, biased, or just wrong sometimes. So, it doesn’t treat the expert’s answer as absolute truth. It treats the answer as a noisy signal.
- Analogy: If the expert says "Yes, there is a tunnel," CaPE thinks, "Okay, that’s helpful, but maybe they’re 80% sure. I’ll adjust my map slightly, not completely."
The Smart Questioning (Expected Information Gain):
This is the heart of the paper. CaPE doesn’t ask random questions. It calculates which question will reduce its confusion the most.
- Analogy: Imagine you have two tunnels you’re unsure about:
  - Tunnel A: You are 99% sure it exists. Asking about it won’t change your mind much.
  - Tunnel B: You are 50/50 unsure. Asking about this will either confirm it or rule it out, drastically changing your map.
- CaPE uses a metric called Expected Information Gain (EIG). It picks the question that maximizes the "aha!" moment. It asks about the edges (connections) where the current data is most ambiguous.
The Update (Refining the Sketch):
Once the expert answers, CaPE updates its "rough sketch." It uses a technique called Particle Filtering.
- Analogy: Imagine you have 1,000 different possible subway maps (particles). When the expert answers a question, CaPE looks at all 1,000 maps. It throws away the ones that contradict the expert’s answer and keeps the ones that fit. It then "rejuvenates" the remaining maps by making small tweaks to ensure it hasn’t narrowed down to just one wrong idea.

Why Is This Better?

The paper claims that this active, smart questioning is much more efficient than the old ways.

Speed: You get a clearer map with fewer questions. In their experiments, CaPE reached a high-quality map much faster than random questioning or simple "uncertainty sampling" (just asking about the most confusing things without considering the global structure).
Robustness: Because it models the expert as "noisy," it doesn’t break if the expert makes a mistake. It gracefully handles uncertainty.
Flexibility: It works with any initial data source. Whether you got your initial guess from a simple statistical model or a complex AI, CaPE can refine it.

Real-World Tests

The authors tested this on three things:

Synthetic Data: Made-up subway maps where they knew the truth. CaPE found the truth faster.
Protein Signaling (Sachs Dataset): A biological network of 11 proteins. Using only observational data (no experiments), CaPE used expert feedback to sharpen the map of how these proteins influence each other.
Gene Perturbation (CausalBench): A larger network of 50 genes. Here, they used experimental data to create a "ground truth" oracle. CaPE successfully identified the correct causal links with fewer queries than baseline methods.

Summary in a Nutshell

CaPE is a framework that turns causal discovery into a dialogue rather than a monologue.

Old Way: "Expert, give me all your rules. I will force my data to fit them."
CaPE Way: "I have a fuzzy map based on data. Which specific connection is most confusing to me? Expert, what do you think about that specific link? Okay, thanks. Now my map is clearer. What’s the next most confusing link?"

By asking the right questions at the right time and respecting that experts can be wrong, CaPE helps scientists build accurate causal maps using less expert time and fewer assumptions.

Technical Summary: Causal Preference Elicitation (CaPE)

Problem Statement
Causal structure learning from observational data is inherently difficult due to the existence of Markov equivalent directed acyclic graphs (DAGs), multi-modal likelihood surfaces, and finite, noisy data. While domain experts possess qualitative knowledge about plausible mechanisms, existing methods for incorporating this knowledge are typically static and coarse, treating expert input as hard constraints or fixed priors. These approaches often ignore expert uncertainty, systematic bias, and the scarcity of expert time, failing to actively query for the most informative judgments. The paper addresses the challenge of integrating expert knowledge into causal discovery in a sequential, information-efficient manner, specifically in settings where experimental interventions are expensive, ethically constrained, or unavailable.

Methodology
The authors propose CaPE (Causal Preference Elicitation), a Bayesian framework for expert-in-the-loop causal discovery. The method operates on an initial observational posterior $q_0(W | X)$ over weighted DAGs, obtained from any black-box causal discovery method.

Expert Likelihood Model: CaPE models expert judgments as noisy categorical responses. For any ordered pair of variables $(i, j)$ , the expert provides a label $Y_{ij} \in \{0, 1, 2\}$ , corresponding to edge $j \to i$ , edge $i \to j$ , or no direct edge. This is modeled via a hierarchical logistic likelihood $p_\theta(Y | W)$ :
- Local Direction Scores: A score $s_{i \to j}(W)$ is computed based on edge weights and optional structural features (e.g., v-structure support, cycle risk).
- Edge Existence: The probability of an edge existing is modeled via a logistic function of the absolute evidence between $i$ and $j$ .
- Orientation: Conditional on existence, the probability of orientation $i \to j$ is modeled via a logistic function of the relative direction scores.
- This structure allows the model to capture expert reliability and decisiveness through hyperparameters $\beta_{edge}$ and $\beta_{dir}$ .
Posterior Inference: The posterior over DAGs is maintained using a particle-based approximation (Sequential Monte Carlo).
- Initialization: Particles $\{W^{(s)}\}$ are sampled from the initial observational posterior $q_0(W | X)$ .
- Update: Upon receiving an expert response $Y_{ij}$ , particle weights are updated via importance reweighting using the expert likelihood.
- Resampling and Rejuvenation: To prevent particle degeneracy, resampling is triggered when the Effective Sample Size (ESS) falls below a threshold. A lightweight Metropolis-Hastings rejuvenation kernel is applied to restore diversity while strictly enforcing acyclicity.
Query Selection: The framework employs an active learning strategy to select which variable pair $(i, j)$ to query next.
- Candidate Screening: To reduce computational cost, the algorithm first screens for the top- $k$ pairs with the highest posterior edge-existence uncertainty.
- Acquisition Function: The final query is selected by maximizing the Expected Information Gain (EIG). The EIG is formulated as the mutual information between the latent DAG $W$ and the expert response $Y_{ij}$ , equivalent to a Bayesian Active Learning by Disagreement (BALD) objective. This metric prioritizes queries that maximize the reduction in global posterior uncertainty (epistemic uncertainty) while accounting for the intrinsic noise of the expert model (aleatoric uncertainty).

Key Contributions

Framework: A general probabilistic framework for causal preference elicitation that combines arbitrary observational posteriors with sequential expert feedback.
Expert Model: A flexible hierarchical logistic model for three-way categorical expert judgments that accounts for noise and uncertainty.
Inference: A particle-based Bayesian updating scheme that integrates expert responses via importance reweighting, ESS-triggered resampling, and MCMC rejuvenation while enforcing DAG acyclicity.
Acquisition Policy: A tractable BALD-style EIG objective for query selection that operates on low-dimensional categorical distributions, enabling efficient active learning on DAG space.
Theoretical Guarantees: Proof of identifiability under non-adversarial expert feedback, showing that the posterior concentrates on the true graph (up to Markov equivalence) as queries are made.

Results
Experiments were conducted on synthetic graphs, the Sachs protein signaling dataset (observational only), and the CausalBench human gene perturbation benchmark.

Synthetic Data: CaPE outperformed random querying, uncertainty sampling, and a static EIG baseline in reducing predictive entropy, Structural Hamming Distance (SHD), and improving Expected True Class Probability (ETCP).
Robustness: The method remained competitive under various misspecifications of the expert model, including heterogeneous reliability, directional bias, and high noise levels.
Sachs Dataset: With only 40 expert queries, CaPE significantly reduced SHD and improved orientation F1 scores compared to baselines, achieving results comparable to strong score-based methods using only observational data.
CausalBench: On a 50-node gene perturbation benchmark, CaPE demonstrated monotonic gains in directed AUPRC and Top-K precision, outperforming baselines at moderate to large query budgets.
Prior Sensitivity: Initializing with a DAG-GFlowNet prior further improved performance, demonstrating the modularity of the framework.

Significance
The paper claims that CaPE provides a principled approach to transforming qualitative expert intuitions into a budgeted sequence of targeted questions. It highlights a complementary regime to interventional experimental design, where structural uncertainty is reduced through targeted expert interaction rather than additional experimental data. The framework is significant for domains where interventions are costly or constrained, offering a way to improve the efficiency and transparency of scientific hypothesis generation by closing the loop between probabilistic causal inference and expert human judgment.