📊 statistics

Cross-validating causal discovery via Leave-One-Variable-Out

This paper introduces a "Leave-One-Variable-Out" (LOVO) prediction framework that falsifies causal discovery algorithms without ground truth by training models on datasets excluding specific variable pairs and evaluating their ability to accurately predict the omitted relationships, thereby using prediction error as a proxy for causal accuracy.

Original authors: Daniela Schkoda, Philipp Faller, Patrick Blöbaum, Dominik Janzing

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniela Schkoda, Philipp Faller, Patrick Blöbaum, Dominik Janzing

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet corners of data science, researchers are trying to solve a puzzle that has long frustrated scientists: how do we know if a computer has truly figured out the hidden causes behind a pattern of events? This field, known as causal discovery, aims to move beyond simply noticing that two things happen together and instead determine if one actually makes the other happen. For decades, the standard way to test these computer programs has been to compare their answers against a known "ground truth," a pre-existing map of reality that researchers use as a reference. However, in the messy real world, such a perfect map rarely exists. Without it, scientists have struggled to know which methods are reliable and which are merely guessing. This uncertainty has left many practical applications, from medicine to economics, stuck in a state of doubt, unable to trust the complex models built to explain them.

To break this deadlock, a team of researchers from Germany has proposed a new way to test these algorithms without needing a pre-drawn map. Their approach relies on a clever trick of omission. Imagine a scientist studying a group of variables, such as temperature, humidity, and wind speed, all interacting in a complex system. Instead of feeding the computer all the data at once, the researchers deliberately hide one pair of variables from the learning process. They teach the algorithm about the remaining variables, then ask it to predict the relationship between the two hidden ones. If the algorithm has truly understood the underlying causal structure, it should be able to infer how the hidden pair connects, even though it never saw them together during its training. This method, which the authors call "Leave-One-Variable-Out" cross-validation, turns the act of prediction into a test of truth.

The core of their work involves a specific type of logical deduction. When the algorithm learns from the data with one variable missing, it builds a model of how the remaining pieces fit together. The researchers then check if this model contains enough information to reconstruct the missing link. In many cases, the structure of the known relationships forces a specific conclusion about the hidden pair. For instance, if the algorithm determines that variable A influences a third variable B, and that B influences variable C, it can logically deduce how A and C are related, even if it never observed A and C directly. The researchers developed two main ways to perform this test. The first is a general method that works with any causal discovery tool, looking for specific patterns in the graph that indicate whether two variables are directly connected or merely influenced by a common cause. The second method is tailored for specific mathematical models that assume linear relationships, allowing the prediction to work even when a direct link exists between the hidden variables.

To see if this idea holds water, the team ran extensive simulations using computer-generated data. They created thousands of different causal systems and tested how well their method could spot errors. The results were encouraging: when the causal discovery algorithm made mistakes in its initial learning phase, the error in the "missing link" prediction grew significantly. In other words, the prediction error acted as a reliable alarm bell. If the algorithm's guess about the hidden relationship was far off, it was a strong signal that the entire model was flawed. Conversely, when the prediction was accurate, it suggested the underlying causal map was likely correct. This correlation was observed across different types of algorithms, including those based on linear equations and those using deep learning techniques.

The researchers also compared their new method against a simpler, "causally agnostic" approach that makes no assumptions about cause and effect. This baseline method assumes that the hidden variables are independent unless the data forces them to be otherwise. The study found that the causal method consistently outperformed this baseline, provided the initial model was reasonably accurate. This suggests that the extra step of inferring a causal structure is not just a theoretical exercise but offers a tangible advantage in predicting relationships between variables that have not been observed together. The team acknowledges that the method is not a magic wand; it cannot solve every problem, and there are specific graph structures where the prediction remains impossible. However, for a wide range of scenarios, it provides a rigorous way to falsify, or disprove, a causal model without needing a pre-existing answer key.

By shifting the focus from matching a known truth to testing the internal consistency of a model, this work offers a new path forward for the field. It suggests that the ability to predict unseen relationships is a powerful metric for judging the quality of causal discovery. While the study relies on simulations rather than real-world interventions, the results provide a compelling argument that we can trust our models more when they can successfully fill in the gaps of their own knowledge. This approach does not just ask if a model fits the data; it asks if the model understands the story the data is telling, even when parts of the story are missing.

Technical Summary: Cross-Validating Causal Discovery via Leave-One-Variable-Out

Problem Statement

The paper addresses the critical challenge of benchmarking and falsifying causal discovery algorithms in the absence of ground truth. While extensive evaluations exist on simulated data, researchers often struggle to determine which method performs best for real-world applications due to a lack of reliable benchmarks. Existing "self-compatibility" checks (e.g., Faller et al., 2024) quantify disagreement between algorithm outputs on subsets and the full set but lack a continuous, optimizable metric to decisively reject an algorithm's outcome. Furthermore, relying on interventional data for validation is often impractical due to cost and complexity.

The authors propose a new framework to evaluate causal discovery algorithms by testing their ability to predict statistical relations between variables that were explicitly excluded during the learning phase. This approach, termed Leave-One-Variable-Out (LOVO) cross-validation, aims to determine if a causal model learned on subsets of variables can successfully infer the dependency between a held-out pair $(X, Y)$ using only the marginal datasets $(X, Z)$ and $(Y, Z)$ .

Methodology

Core Concept: LOVO Prediction

Given a set of variables $W = \{X, Y, Z_1, \dots, Z_k\}$ , the method involves:

Data Partitioning: Creating two "leave-one-out" datasets: $D_{XZ}$ (containing $X$ and $Z$ ) and $D_{YZ}$ (containing $Y$ and $Z$ ), excluding the pair $(X, Y)$ .
Causal Learning: Applying a causal discovery algorithm separately to $D_{XZ}$ and $D_{YZ}$ to obtain marginal causal graphs $G_X$ and $G_Y$ .
Inference: Using $G_X$ and $G_Y$ to infer the joint causal structure $G$ and subsequently estimate the conditional expectation $E[Y | X = x]$ or the correlation $\rho_{XY}$ without observing $X$ and $Y$ jointly.
Evaluation: Comparing the LOVO prediction against the true joint distribution $P(X, Y)$ (which is available for the purpose of falsification) to calculate a continuous prediction error. This error serves as a metric for the reliability of the causal discovery algorithm.

Theoretical Foundations

The paper establishes that LOVO prediction is not generally possible via probabilistic laws alone (Lemma 1), as marginal distributions $P(X, Z)$ and $P(Y, Z)$ do not uniquely determine $P(X, Y)$ . However, causal models provide the necessary inductive bias to resolve this ambiguity.

The authors propose two primary variants of LOVO predictors:

Graphical Method (Parent Adjustment):
- Applicable to general algorithms outputting Directed Acyclic Graphs (DAGs) or Acyclic Directed Mixed Graphs (ADMGs).
- Edge Exclusion: Lemmas 2 and 3 provide criteria to determine if $X$ and $Y$ are unlinked (i.e., no edge exists between them) based solely on the marginal graphs $G_X$ and $G_Y$ .
- Prediction: If $X$ and $Y$ are unlinked, the union of their parents ( $Z_S$ ) serves as a d-separating (or m-separating) set. The predictor is constructed via parent adjustment:
  $P(y|x) = \sum_{z_S} P(y|z_S)P(z_S|x)$
- This is implemented via a three-step procedure: learning a predictor from $P(Y|Z)$ , generating artificial pairs $(x, \hat{y})$ , and learning the final predictor $P(Y|X)$ .
Model-Specific Method (LiNGAM):
- Tailored for algorithms assuming Linear Non-Gaussian Acyclic Models (LiNGAM).
- If the edge type between $X$ and $Y$ can be identified (even if linked) and specific child-set conditions are met, the structure matrix $\Lambda$ can be uniquely identified from the marginals.
- This allows for the reconstruction of $P(X, Y, Z)$ and the estimation of $E[Y|X]$ even in the presence of direct causal links, provided the model assumptions hold.

Baseline Comparison

To ensure the causal information is genuinely useful, the paper introduces a MaxEnt Baseline Predictor. This predictor assumes $X \perp\!\!\perp Y | Z$ (maximizing entropy subject to marginals) and uses the formula $P_{MaxEnt}(y|x) = \sum_z P(y|z)P(z|x)$ . The LOVO error is evaluated relative to this baseline; a causal algorithm is considered successful if its LOVO error is significantly lower than the baseline error.

Key Contributions

LOVO Cross-Validation Framework: A novel method for falsifying causal discovery algorithms without ground truth by treating the prediction of a held-out variable pair as a test of the model's generalization capability ("out-of-variable generalization").
Theoretical Conditions for Realizability: Formal proofs (Lemmas 2, 3, and Theorems 4, 5) delineating exactly when LOVO prediction is feasible based on marginal graphs, distinguishing between cases where edges can be excluded and cases where structural equation parameters can be recovered.
Continuous Evaluation Metric: Unlike previous discrete disagreement metrics, LOVO provides a continuous prediction error, allowing for the optimization of causal discovery algorithms and the establishment of rigorous rejection baselines.
Deep Learning Extension: The authors demonstrate that deep learning architectures (specifically transformer-based encoders) can be adapted to learn causal representations from marginal datasets to predict correlations, outperforming the MaxEnt baseline without explicit parametric assumptions.

Experimental Results

The authors conducted extensive simulations to validate the method:

Feasibility: In random graphs (Erdős–Rényi DAGs and ADMGs), the graphical lemmas successfully identified unlinked pairs in the vast majority of cases, making LOVO prediction realizable.
Performance: When provided with true marginal graphs, both the Parent Adjustment and LiNGAM-based LOVO predictors significantly outperformed the MaxEnt baseline.
Correlation with Accuracy: When applied to causal discovery algorithms (DirectLiNGAM and Repetitive Causal Discovery), the LOVO prediction error was found to be positively correlated with:
- The frequency of misidentified edges (specifically, misidentifying linked pairs as unlinked).
- The Structural Hamming Distance (SHD) of the learned graphs.
- The Spearman correlation coefficients were significant, indicating that higher LOVO error reliably signals lower causal discovery accuracy.
Deep Learning: A custom deep learning architecture trained on marginal datasets successfully predicted correlations and outperformed the baseline, suggesting it implicitly learns causal representations.

Significance and Claims

The paper claims that LOVO cross-validation offers a rigorous, ground-truth-free mechanism to evaluate causal discovery algorithms. By demonstrating that the prediction error correlates with the accuracy of the learned causal structure, the authors argue that LOVO can serve as a reliable benchmark for selecting causal discovery methods in practical applications where ground truth is unavailable.

The work emphasizes that causal models enable "interpolation" of dependencies between variable pairs that have not been observed together, a capability distinct from standard statistical learning. The authors position LOVO not as a replacement for all validation methods but as a necessary tool for the "self-check" of causal hypotheses, particularly in scenarios where interventional data is inaccessible. The paper remains modest, noting that LOVO prediction is not always possible (e.g., in small graphs or specific structural configurations) and that the method relies on the faithfulness assumption and the correctness of the underlying causal discovery algorithm's output format (DAGs/ADMGs).