🤖 machine learning

Large Causal Models for Temporal Causal Discovery

This paper introduces Large Causal Models (LCMs), a pre-trained foundation model paradigm that overcomes the limitations of traditional dataset-specific approaches by scaling to higher variable counts and achieving superior, fast inference performance in temporal causal discovery across diverse synthetic and realistic benchmarks.

Original authors: Nikolaos Kougioulis, Nikolaos Gkorgkolis, MingXue Wang, Bora Caglayan, Dario Simionato, Andrea Tonon, Ioannis Tsamardinos

Published 2026-08-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nikolaos Kougioulis, Nikolaos Gkorgkolis, MingXue Wang, Bora Caglayan, Dario Simionato, Andrea Tonon, Ioannis Tsamardinos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of looking for fingerprints or footprints, you are looking for cause and effect in a stream of data. This is the world of causal discovery, a branch of science dedicated to figuring out what makes what happen. Usually, when scientists study a specific system—like the weather, a stock market, or the human brain—they have to build a custom detective tool from scratch for that one specific case. It's like hiring a new private investigator for every single crime scene, training them from zero, and hoping they get it right. This is slow, expensive, and doesn't help the next case.

The big idea behind this paper is to build a "super-detective" that has already seen millions of different crime scenes. In the world of artificial intelligence, these are called Foundation Models. Think of them like a genius student who has read every book in the library before taking a test. Instead of studying for a specific exam, they use their vast, general knowledge to solve new problems instantly. The paper focuses on temporal causal discovery, which is just a fancy way of saying "figuring out cause and effect over time." It asks: If event A happened yesterday, did it cause event B to happen today? The authors want to know if we can train one massive, smart AI to do this for any time-based data, from energy grids to heartbeats, without needing to retrain it every time.

The authors, a team from the University of Crete and Huawei, propose a new kind of super-detective called a Large Causal Model (LCM). In the past, attempts to build these "super-detectives" for time-series data were like trying to teach a toddler to solve quantum physics; they worked okay on tiny, simple puzzles but fell apart when the problems got bigger or more complex. The researchers found that these failures weren't because the AI wasn't smart enough, but because the "textbooks" they were studying were too boring and repetitive. They were mostly made up of fake, synthetic data that didn't look like the real world.

To fix this, the team created a massive, diverse training library. They mixed in millions of examples of fake data with thousands of realistic, real-world time-series samples (like weather patterns and power consumption). They then trained their LCM on this huge mix. The results were impressive: the new model could look at a time-series it had never seen before and instantly guess the causal structure with high accuracy. It didn't need to run slow, complicated calculations for each new dataset; it just made a single, lightning-fast pass through the data.

The paper suggests that these Large Causal Models are a game-changer. In tests, they performed as well as, or even better than, the old, specialized methods that take much longer to run. Crucially, they showed they could handle "zero-shot" scenarios, meaning they could solve problems in areas they were never explicitly trained on, just by relying on the broad patterns they learned during training. The authors measured this using a standard score called AUC (Area Under the Curve), where higher is better, and their models consistently hit scores near 0.98 or 0.99 on many realistic benchmarks, beating the competition.

However, the paper is careful to note that this isn't magic. The model still relies on certain rules, like assuming there are no hidden "ghost" variables messing up the data. If those rules are broken, the model might give a confident but wrong answer. But for the vast majority of cases where data is clean and follows standard time-based rules, this approach suggests we can finally move away from building a new tool for every single dataset. Instead, we can have one powerful, pre-trained brain that solves causal mysteries in a flash, potentially speeding up discoveries in fields ranging from biology to economics. The authors have even made their code and models available for others to try, inviting the rest of the scientific community to join the investigation.

Technical Summary: Large Causal Models for Temporal Causal Discovery

1. Problem Statement

Causal discovery (CD) for temporal data aims to recover the causal structure (lagged causal graphs) from multivariate time series. Traditional methods, whether constraint-based (e.g., PCMCI), score-based (e.g., DYNOTEARS), or functional-model-based (e.g., VARLiNGAM), typically follow a dataset-specific paradigm. They require fitting a new model for each individual dataset, often relying on conditional independence testing or combinatorial search. These approaches suffer from poor scalability with increasing input dimensionality and variable counts. Furthermore, existing neural approaches to CD often fail to generalize across heterogeneous systems, particularly in zero-shot settings, and prior attempts at "Large Causal Models" (LCMs) have been limited to small variable counts and narrow synthetic training distributions, leading to performance degradation as dimensionality increases.

The core challenge addressed is the scarcity of large, diverse time-series datasets with ground-truth causal graphs, which has historically prevented the training of scalable foundation models for temporal CD.

2. Methodology

2.1 Large Causal Models (LCMs)

The authors propose Large Causal Models (LCMs), a class of pre-trained neural architectures designed as foundation models for temporal causal discovery. Unlike traditional methods that perform iterative search per dataset, LCMs learn an amortized approximation of the mapping from time-series samples to causal graphs.

Architecture: The implemented LCM follows a convolution-enhanced Transformer encoder architecture.
- Input Embedding: Multivariate time series are normalized, padded with Gaussian noise (to handle variable lengths without zero-artifacts), and projected via 1D Convolutional layers to extract local temporal features.
- Encoder Stack: Utilizes multi-head self-attention and feed-forward networks with residual connections. A distillation block (Conv1D + Pooling) is inserted to reduce sequence length for long-horizon processing.
- Training Aids: To enhance inductive bias, lagged cross-correlations are computed from the input and concatenated to the final encoder representations.
- Output Head: A feed-forward network outputs a binary adjacency tensor $\hat{A} \in [0, 1]^{V_{max} \times V_{max} \times \ell_{max}}$ , representing the probability of directed edges $X^i_{t-\ell} \to X^j_t$ .
Loss Function: The model is trained via supervised learning using a composite loss:
1. Edge Prediction Loss: Binary cross-entropy between the predicted adjacency tensor and the ground truth.
2. Correlation Regularization (CR): A weighted mean squared error term aligning predictions with observed lagged cross-correlations in the input data, encouraging confident predictions to align with empirical dependencies.

2.2 Data Curation and Training Strategy

A critical contribution is the construction of a Large-Scale Corpus combining diverse data sources to enable robust zero-shot generalization:

Synthetic Data: Generated via parametric random families (e.g., Erdős–Rényi graphs) with varying variables, lags, densities, and functional mechanisms (linear and nonlinear).
Realistic Data: Derived from real-world time series (energy, weather, transportation) using a method that learns a Temporal Structural Causal Model (TSCM) from observed data to synthesize "causal twins" with known ground truth.
Semi-Synthetic Data: Mechanistic simulators (fMRI, Kuramoto oscillators) used exclusively for Out-of-Distribution (OOD) evaluation.
Mixing Strategy: Experiments show that training on a mixture of 80% synthetic and 20% realistic data yields optimal performance, balancing structural diversity with real-world fidelity.

3. Key Contributions

Formalization of LCMs: The paper formalizes LCMs as foundation models for temporal CD, replacing dataset-specific algorithms with a pretraining paradigm capable of handling heterogeneous data distributions.
Scaling Analysis: The authors demonstrate that previous scaling failures in causal discovery models were due to insufficient training distribution diversity rather than architectural limitations. By training on a large, heterogeneous corpus, LCMs scale effectively to higher variable counts (up to $V_{max}=12$ in experiments) and deeper architectures.
Large-Scale Benchmarking: The paper curates a substantial collection of time-series and causal graph pairs (275k instances) suitable for both training LCMs and benchmarking existing CD methods.
Zero-Shot Generalization: The trained models achieve competitive or superior accuracy compared to classical baselines in zero-shot settings (unseen datasets), enabling fast, single-pass inference without retraining.

4. Experimental Results

Experiments were conducted on synthetic, semi-synthetic, and realistic benchmarks, comparing LCMs (ranging from 2.5M to 24M parameters) against baselines like PCMCI, DYNOTEARS, and VARLiNGAM.

Performance: LCMs consistently match or outperform classical baselines across in-distribution and out-of-distribution (OOD) settings. For instance, on the Synthetic_2 benchmark, LCMs achieved AUC scores around 0.909, significantly outperforming PCMCI (0.800) and DYNOTEARS (0.605).
Generalization: Models trained on mixed synthetic/realistic data showed statistically significant improvements in OOD performance (e.g., on fMRI and Power/Climate datasets) compared to models trained solely on synthetic data.
Efficiency: LCMs offer a substantial runtime advantage. While traditional methods exhibit superlinear scaling with variable count and lag, LCM inference is effectively independent of input dimensionality due to the constant-time forward pass.
Ablation: The inclusion of training aids (cross-correlations) and correlation regularization significantly improved performance.

5. Significance and Claims

The paper positions LCMs as a promising foundation-model paradigm for temporal causal discovery. The primary significance lies in demonstrating that:

Data-Centric Scaling: Robustness and scalability in causal discovery are achievable through data-centric training on large, diverse corpora, rather than solely relying on architectural complexity.
Efficiency: LCMs enable fast, single-pass causal discovery, removing the computational bottleneck of iterative search required by classical methods.
Practicality: By leveraging pre-trained models, LCMs offer a practical alternative for scientific domains requiring rapid causal inference without the need for costly dataset-specific retraining or randomized control trials.

The authors note that LCMs operate under specific causal assumptions (causal sufficiency, stationarity, no contemporaneous effects) and do not detect violations of these assumptions. Future work is identified as relaxing these assumptions to handle latent confounders and contemporaneous effects, as well as scaling to higher-dimensional systems.