🤖 AI

CausalMoE: A Billion-Scale Multimodal Foundation Model for Granger Causal Discovery with Pattern-Routed Heterogeneous Experts

CausalMoE is a billion-scale multimodal foundation model that leverages a pattern-routed mixture of heterogeneous experts and causality-aware self-attention to overcome the limitations of traditional "one-size-fits-all" approaches, achieving state-of-the-art Granger causal discovery with improved generalization and interpretability by integrating textual and visual priors.

Original authors: Bo Liu, Di Dai, Jingwei Liu, Jiarui Jin, Xiaocheng Fang, Guangkun Nie, Hongyan Li, Shenda Hong

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bo Liu, Di Dai, Jingwei Liu, Jiarui Jin, Xiaocheng Fang, Guangkun Nie, Hongyan Li, Shenda Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out who is influencing whom in a chaotic, noisy room full of people talking at once. Maybe a shout from the kitchen causes a reaction in the living room, or maybe two people are just laughing at the same joke without talking to each other. This is the challenge of Granger Causal Discovery: trying to find the true "cause-and-effect" chains in complex data over time.

For a long time, computers tried to solve this by using a "one-size-fits-all" approach. They assumed the whole room followed the same set of rules. But in the real world, the rules change. Sometimes the kitchen is quiet, sometimes it's chaotic. When a computer tries to apply one single rule to a room that keeps changing its mood, it gets confused. It starts seeing connections that aren't there (like thinking the toaster caused the dog to bark just because they happened at the same time).

Enter CausalMoE, a new "billion-scale" AI model that acts like a super-smart detective with a team of specialized experts. Here is how it works, broken down into simple parts:

1. The "Chameleon" Strategy: Pattern-Routed Experts

Instead of using one giant brain to analyze the whole room, CausalMoE breaks the data into small chunks (like taking snapshots of the room every few seconds). It then uses a smart traffic controller (called a "Pattern-Routed Mixture of Heterogeneous Experts") to look at each snapshot.

The Analogy: Imagine a hospital emergency room. If a patient comes in with a broken leg, you don't send them to a cardiologist; you send them to an orthopedist.
How it works: CausalMoE looks at a chunk of data and asks, "What kind of pattern is this? Is it a steady rhythm? Is it a chaotic spike? Is it a seasonal trend?" It then instantly routes that specific chunk to the specialist expert best suited to handle it.
- One expert is great at spotting seasonal trends (like the weather changing).
- Another is an expert in fast, chaotic changes (like stock market crashes).
- Another is a semantic expert that reads the "story" behind the numbers.
- Another is a visual expert that looks at the "shape" of the data.

By letting the right expert handle the right situation, the model stops mixing up different types of causes, which prevents it from making false connections.

2. The "Multimodal" Superpower: Reading Between the Lines

Most old models only looked at the raw numbers (the temperature, the stock price, the heart rate). CausalMoE is different because it is multimodal. It doesn't just look at the numbers; it translates them into other languages to get a better understanding.

The Text Translator: It turns a chunk of numbers into a text prompt (like a short story or a news headline) and asks a massive Language Model (LLM) to read it. This helps the AI understand the "context" or the "vibe" of the data.
The Visual Artist: It turns the numbers into a picture (like a line graph) and asks a Vision-Language Model (VLM) to look at it. This helps the AI see the "shape" of the data, like a sharp peak or a smooth curve, which might be hard to see in a spreadsheet.

Why does this help? Imagine trying to guess why a car stopped. If you only look at the speedometer (the number), you might think it ran out of gas. But if you look at a picture of the car (visual) and read a note saying "flat tire" (text), you get the real answer. CausalMoE uses these extra "clues" to figure out the true cause, even when the numbers are messy or scarce.

3. The "Truth Detector": Causality-Aware Attention

Once the experts have done their job, the model needs to draw the final map of who caused what. It uses a special tool called Causality-Aware Self-Attention.

The Analogy: Think of a group of friends trying to decide who started a rumor. Instead of everyone talking to everyone at once, this tool forces the AI to ask, "If I change this person's story, does that person's story change?"
It forces the AI to be very strict and only draw lines between variables that truly influence each other, ignoring the noise. This results in a clean, simple map (a "sparse graph") rather than a tangled mess of fake connections.

Why is this a Big Deal?

The paper claims that CausalMoE is a massive leap forward for two main reasons:

It works when data is scarce: Usually, AI needs thousands of examples to learn. CausalMoE can figure out the rules with very few examples (a "few-shot" setting) because it uses the "world knowledge" from its text and image experts to fill in the gaps. It's like a detective who can solve a crime with just a few clues because they already know how the criminal usually operates.
It handles chaos: Real-world data is messy and changes its behavior. Because CausalMoE routes different chunks to different experts, it doesn't get confused when the rules change. It adapts on the fly.

In summary: CausalMoE is a billion-parameter AI that stops trying to force a single rule on a changing world. Instead, it acts like a team of specialists who translate numbers into stories and pictures, route the data to the right expert, and then draw a clear, honest map of cause and effect. The paper shows it beats all previous methods on standard tests, especially when there isn't much data to work with.

Technical Summary: CausalMoE

Problem Statement

Granger Causal Discovery (GCD) is a fundamental paradigm for analyzing temporal dependencies in complex systems. However, existing neural GCD methods predominantly rely on a "one-size-fits-all" Uniform Distribution Modeling (UDM) strategy. This approach assumes a homogeneous data-generating process, treating all temporal segments as samples from a single underlying distribution. In real-world time series, this assumption fails to account for distribution shifts and dynamic regime changes that occur even at the granularity of short temporal patches. Consequently, UDM-based methods often conflate heterogeneous regimes, leading to entangled representations and spurious causal graphs where genuine causality is diluted.

Furthermore, current GCD methods are strictly unimodal, relying exclusively on numerical observations. In complex systems, numerical correlations alone are often insufficient for causal identification, as semantic context (textual descriptions, event annotations) or visual states often act as critical confounders or disambiguating signals. While recent Time Series Foundation Models (TSFMs) address data scarcity through large-scale pretraining, they optimize for prediction rather than causal fidelity, often absorbing structural variations into latent representations and eroding the invariances required for causal discovery.

Methodology: CausalMoE

To address these limitations, the authors propose CausalMoE, a billion-scale multimodal Granger causal foundation model. The architecture departs from static modeling assumptions by explicitly modeling patch-level temporal heterogeneity and integrating multimodal priors. The framework consists of three core modules:

1. Multimodal Patch Encoding

The input multivariate time series is divided into patches. To leverage semantic and visual priors, each patch is transformed into three modalities:

Temporal: Raw numerical patches.
Textual: Patches are wrapped into structured prompts containing dataset context, historical data, statistical features (e.g., trends, medians), and task instructions. These are processed by a frozen Large Language Model (LLM).
Visual: Patches are resized and normalized into images using bilinear interpolation. These are processed by a frozen Vision-Language Model (VLM).
Crucially, these modalities are deterministic views of the observed historical patches, not additional exogenous variables, ensuring they do not introduce new confounders.

2. Pattern-Routed Mixture of Heterogeneous Experts (MoHE)

To handle distribution shifts, CausalMoE introduces a Patch-Specific Pattern Routing (PSPR) module. This module uses subspace clustering to detect latent temporal patterns (e.g., seasonality, trends, spectral variations) and dynamically routes each patch to the most suitable domain expert.
The MoHE architecture comprises four specialized experts:

Semantic Expert: Encodes prompt-level semantic regularities (trends, noise patterns) using the frozen LLM.
Multimodal Expert: Captures shape-level visual structures (peak sharpness, envelope morphology) using the frozen VLM.
Temporal Frequency Expert: Combines time-domain features with Fourier-domain representations to capture stationary spectral patterns.
Multiscale Temporal Expert: Aggregates downsampled representations to model non-stationary trends and hierarchical seasonal patterns.
A gating network selects the top- $k$ experts for each patch, enabling the decoupling of regime-specific mechanisms from shared dynamics.

3. Causality-Aware Self-Attention (CASA) and Graph Recovery

To ensure interpretable graph recovery, CausalMoE employs a Causality-Aware Self-Attention (CASA) mechanism. Unlike standard self-attention that operates across time steps, CASA operates across variables. It replaces standard projection matrices with variable-level projections ( $\omega_q, \omega_k, \omega_v \in \mathbb{R}^{D \times D}$ ) to compute attention weights that explicitly encode variable-to-variable influence.
The causal graph is inferred via proximal optimization. A regularization term is applied to the value projection weights to promote sparsity in the causal adjacency matrix. The optimization uses proximal gradient descent with group soft-thresholding to solve the non-convex objective, yielding sparse, interpretable Granger causal graphs.

Key Contributions

CausalMoE Architecture: The proposal of a billion-scale multimodal foundation model that explicitly models patch-level temporal heterogeneity. It utilizes a Pattern-Routed Mixture of Heterogeneous Experts to dynamically route patches to specialized experts, decoupling regime-specific causality from shared dynamics.
Multimodal Integration: The first integration of LLMs and VLMs into the GCD loop. By aligning numerical signals with textual and visual priors, the model regularizes causal estimation in complex scenarios where numerical data alone is ambiguous.
Causality-Aware Mechanism: The design of the CASA mechanism and proximal optimization strategy to recover sparse, interpretable Granger causal graphs directly from the routed multimodal representations.

Experimental Results

Extensive experiments were conducted on five benchmark datasets: synthetic VAR and Lorenz-96 systems, simulated fMRI data, and gene expression datasets (DREAM-3 and DREAM-4).

Supervised Performance: CausalMoE establishes a new state-of-the-art (SOTA) across all fully supervised benchmarks. It significantly outperforms traditional methods (e.g., GC, PCMCI) and recent neural approaches (e.g., NGC, CUTS, JRNGC), particularly on complex, nonlinear datasets like Lorenz-96 and DREAM-4.
Few-Shot Generalization: In few-shot settings (training on 1%–10% of data), CausalMoE demonstrates exceptional robustness. It matches the performance of baselines trained on full data while using only a fraction of the observations. For instance, on DREAM-4, it achieves an AUROC > 0.6 with only 5% of the training data, whereas conventional neural methods degrade sharply.
Ablation Studies: Removing the pattern router, multimodal transformation, or CASA mechanism consistently degrades performance, confirming that each component is necessary for disentangling regimes, regularizing dependencies, and extracting sparse causal structures.
Efficiency: While larger foundation backbones improve performance, the lightweight variant of CausalMoE maintains strong causal discovery capabilities with substantially reduced training costs, as the foundation models serve as frozen feature extractors.

Significance and Claims

The paper claims that CausalMoE represents a paradigm shift from static, one-size-fits-all causal modeling to adaptive, regime-aware causal modeling. By addressing the dichotomy between static modeling assumptions and dynamic real-world heterogeneity, CausalMoE enables reliable causal inference in data-scarce and complex scenarios where traditional methods fail.

The authors emphasize that their approach does not merely scale up existing models but fundamentally rethinks the causal discovery pipeline by:

Explicitly modeling temporal heterogeneity rather than smoothing it out.
Leveraging semantic and visual priors to disambiguate causal links unidentifiable from numerical data alone.
Providing a practical framework for "causal-discovery-as-a-service" that is robust to distribution shifts and limited data availability.

The work is positioned as a foundational step toward multimodal causal discovery, though the authors note limitations regarding the reliance on predictive dependencies (Granger causality) rather than interventional causality, and the potential for biases inherited from pre-trained foundation models.