🤖 machine learning

DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series

The paper introduces DoTime, a scalable and theoretically grounded synthetic benchmark generator for multivariate temporal structural causal models that addresses the lack of interventional and counterfactual time series data by providing open tools, evaluation suites with exact ground truth, and empirical evidence that interventional training improves causal direction accuracy over observational models.

Original authors: Dennis Thumm, Billy Tim Anthony, Ying Chen

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dennis Thumm, Billy Tim Anthony, Ying Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but the only clues you have are a pile of old, dusty photos. You can see what happened in the past, but you can never see what would have happened if someone had made a different choice. This is the daily struggle for scientists who study "causality"—the art of figuring out what causes what. In the real world, we often just watch things happen (observational data), like watching a plant grow in the sun. But to truly understand cause and effect, we need to ask "What if?" questions (counterfactuals). What if we watered the plant instead? What if we turned off the lights? To answer these "What if?" questions, we usually need to run experiments, but in fields like medicine or climate science, running experiments can be dangerous, expensive, or even impossible.

So, scientists have started building "simulators"—digital worlds where they can run experiments safely. They create fake data that follows the rules of cause and effect, train their computer brains (AI models) on this fake data, and then hope those brains are smart enough to solve real-world mysteries. The big question is: How do we know if these digital test worlds are good enough? If the simulator is too simple, the AI learns the wrong lessons. If it's too messy, we can't tell if the AI is actually smart or just lucky. We need a perfect, fair, and challenging playground to test these AI detectives before they go out into the real world.

Enter DoTime, a new tool created by researchers Dennis Thumm, Billy Tim Anthony, and Ying Chen. Think of DoTime as the ultimate "flight simulator" for time-traveling detectives. It is a computer program that generates millions of fake, complex stories about how things change over time, complete with a secret "answer key" that tells you exactly what would happen if you changed the plot.

The paper introduces DoTime as a way to fix a major gap in how we test AI. Most existing test worlds only show the AI what did happen, not what could have happened. DoTime is different because it lets the AI practice "interventional" thinking. It creates scenarios where you can say, "Okay, in this story, let's pretend we gave the patient a different medicine," and the simulator instantly rewrites the future to show the result, while keeping everything else exactly the same. It even has a special "shared noise" mode, which is like having two identical twins: one lives their normal life, and the other gets a specific treatment. Because they are twins, any difference in their lives is guaranteed to be caused by that treatment, not by random luck.

The researchers used DoTime to test a specific idea: Does training an AI on these "What if?" stories make it better at solving real problems than just watching normal stories? They set up a fair race between two AI models of the exact same size. One model was trained only on normal, observational stories (watching what happens). The other was trained on the "What if?" intervention stories. The results were clear and measurable: the model trained on the "What if?" stories consistently got the direction of the cause-and-effect right more often than the one that only watched. In their simulations, this "interventional training" gave the AI a measurable advantage, improving its accuracy by about 8 to 9 percentage points compared to its observational twin.

However, the authors are careful not to claim they have solved the world's problems. They explicitly rule out the idea that the AI just got lucky or that it simply memorized the answers. They tested the AI on different types of stories, different lengths of time, and even on real-world data (like wind tunnel experiments and drug reactions) to see if the skills transferred. While the AI showed it could handle real data, the results were mixed; it was good at predicting the general "level" of an outcome but sometimes struggled to track the tiny, fast wiggles in the data. This suggests that while DoTime is a powerful new training ground, the AI is still learning and isn't perfect yet.

In short, DoTime is a massive, open-source playground that lets scientists build better "time-travel" detectives. It proves that teaching an AI to imagine different futures makes it smarter at understanding cause and effect, but it also shows us exactly where these digital brains still stumble. It's a tool for building better models, not a magic wand that fixes everything overnight.

Technical Summary: DoTime

Problem Statement

Current benchmarks for causal inference in time series are predominantly observational, small-scale, or restricted to specific domains. This leaves a critical gap in evaluating interventional and counterfactual estimation, which are essential for high-stakes applications like healthcare, policy evaluation, and climate science. Existing tools suffer from three main limitations:

Observational Bias: Generators like Dream3 and NetSim focus on structural discovery rather than effect estimation ($p(Y|do(A))$), training models on data that never exposes them to interventions.
Scalability Issues: Real-world interventional datasets (e.g., Causal Chambers) are too small to train foundation models.
Static Limitations: Static benchmarks (ACIC, IHDP) ignore temporal dynamics, while recent dynamic benchmarks (CausalDynamics) focus on graph recovery from observational data rather than interventional effect estimation.

Consequently, models trained to answer $do(\cdot)$ queries are often fitted and scored on purely observational data ( $p(Y|A)$ ), creating a skew that fails to validate true causal reasoning capabilities.

Methodology: The DoTime Framework

DoTime is an open, scalable, and theoretically grounded generator of multivariate Temporal Structural Causal Models (TSCMs). It operates via a pipeline that samples structural priors, simulates trajectories under both observational and interventional conditions, and provides exact ground-truth targets.

Core Components

Temporal Structural Causal Models (TSCMs):
- Defined by a graph collection $\mathcal{G} = (G_0, G_{1:K})$ , where $G_0$ is an instantaneous DAG and $G_{1:K}$ are lagged adjacencies.
- Dynamics follow $X^i_t = f_i(Pa^0_i(X_t), Pa^{1:K}_i(X_{t-1:t-K})) + \varepsilon^i_t$ .
- Interventions override the mechanism $f_i$ for targeted nodes over specific time windows.
Ground Truth Levels (Pearl's Hierarchy):
- Level 2 (Interventional): The intervened arm draws fresh exogenous noise from the prior $P(\varepsilon)$ .
- Level 3 (Counterfactual): The intervened arm reuses the same noise stream $\varepsilon$ that generated the observational trajectory. DoTime's continuous-time suite generates Level 3 counterfactuals by construction; discrete-time suites provide Level 2 interventional twins.
Key Extensions Beyond Prior Generators:
- Intervention Types: Supports hard ( $do(A_t=v)$ ), soft (additive shifts), and time-varying interventions.
- Continuous-Time Windows: Generalizes single-step interventions to windows $do(A[t_0, t_1]=v)$ using Euler–Maruyama integration of underlying SDEs. This allows for irregular observation grids (jittered or Poisson).
- Counterfactual Modes: Includes a "positivity guard" that clips intervention values to the observed support ( $\hat{\mu} \pm 3\hat{\sigma}$ ), distinguishing between extrapolative and in-support counterfactuals.
- Regime-Switching: Implements sticky-Markov regime chains over SCM parameters, generalizing Interrupted Time Series (ITS) designs to stochastic switches and multiple regimes.
Graph and Mechanism Priors:
- Graphs: $G_0$ is Erdős–Rényi with acyclicity guaranteed by construction (upper-triangular mask). Lagged edges follow geometric decay.
- Mechanisms: Uniformly sampled from a diverse set including linear, tanh, sin, cos, ReLU, and compositions.
- Noise: Scales sampled from shifted exponentials.

Key Contributions

The paper releases the DoTime ecosystem, comprising:

The dotime PyPI Package: Contains the generator, suite loaders, and evaluation harness.
Four Frozen Evaluation Suites:
- dot-Identifiability-v1: 10,800 trajectories across 8 named identification structures (e.g., Back-Door, Front-Door, Instrumental Variable, Unobserved Confounder).
- dot-RegimeSwitch-v1: 10,000 trajectories with regime densities of 2, 3, or 5.
- dot-Continuous-v1: 10,000 trajectories with continuous-time intervention windows and varying query offsets.
- dot-Generic-100k: 100,000 trajectories from the full diverse prior, sized for pre-training foundation models.
Reference Baselines: Implementations of heuristics (TrajMean, AR1), classical methods (VAR, BackDoorOLS, IV2SLS), and a Do-Over-Time-PFN (Prior-Fitted Network) trained on both interventional and observational contexts.

Results

The paper evaluates the Do-Over-Time-PFN against baselines and an observational twin of identical capacity.

Metric Limitations: On these nonlinear TSCMs, standard RMSE metrics are near-saturated (models cluster around means). The paper argues that direction accuracy (predicting the sign of the effect) is the discriminating signal for causal quality.
Interventional vs. Observational Training:
- A model trained on interventional data ( $PFN_{int}$ ) consistently outperforms an identical model trained only on observational data ( $PFN_{obs}$ ) in terms of direction accuracy.
- Gap Magnitude: On the Identifiability suite, $PFN_{int}$ achieves 0.66 accuracy vs. 0.57 for $PFN_{obs}$ (+0.08 gap). On Generic, the gap is +0.04.
- Robustness: This advantage persists across three training seeds, four trajectory lengths ( $T \in \{200, 500, 1000, 2000\}$ ), and three identification structures (Back-Door, Front-Door, IV). The pooled gap is +0.09 with a seed-to-seed standard deviation of only 0.005.
Test-Time Context: The advantage is attributed to interventional pre-training, not merely seeing the intervention value at test time. Observational models (TabPFN, Chronos-2) given the intervention value at test time showed no accuracy gap.
Zero-Shot Transfer: The synthetic-trained PFN transfers to real-world domains (Causal Chambers wind tunnel and Warfarin pharmacokinetics). While it does not track high-frequency micro-oscillations, it successfully recovers the coarse regime relocation (post-intervention operating level) where naive baselines fail.

Significance and Claims

The paper positions DoTime as the first open, scalable benchmark specifically for interventional and counterfactual time series. Its primary claim is a falsifiable hypothesis: Interventional training buys a measurable direction-accuracy advantage over an observational model of identical capacity.

The authors demonstrate that:

Causal Understanding: The benchmark successfully isolates causal understanding at fixed capacity, where traditional regression metrics fail.
Necessity of Synthetic Data: Real-world interventional data is scarce; DoTime provides a principled, theoretically grounded alternative for training foundation models.
Limitations: The authors are modest about the scope. The current suites use $N \le 10$ variables and $K \le 3$ lags (though the generator supports more). The synthetic data does not yet cover trending/seasonal confounders or partial observability beyond hidden confounders. The transfer results to real data are presented as proof-of-concept "regime relocation" rather than competitive dominance over domain-specific tools.

In summary, DoTime provides the infrastructure to move time-series causal inference from structural discovery on observational data to rigorous effect estimation under intervention, validating that training on synthetic interventional data yields models with superior causal reasoning capabilities.