CauKer: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data

The paper introduces CauKer, a novel algorithm that combines Gaussian Process kernel composition with Structural Causal Models to generate diverse, causally coherent synthetic time series, enabling sample-efficient pre-training of classification foundation models that exhibit clear scaling laws across varying dataset sizes and model capacities.

The Gist

Imagine you want to teach a robot to recognize different types of animals just by looking at their footprints in the mud.

The Old Way (Real Data):
Traditionally, to teach this robot, you would need to go out into the wild, collect millions of real footprints from lions, tigers, bears, and elephants. You'd have to spend years gathering them, cleaning them up, and organizing them. It's expensive, slow, and sometimes you just can't find enough rare footprints to teach the robot well.

The New Way (CAUKER):
This paper introduces a new method called CAUKER. Instead of going into the wild, the researchers built a high-tech "Mud Factory" inside a computer. This factory can instantly print out millions of fake footprints that look and feel exactly like real ones, but with a secret superpower: they are perfectly organized to teach the robot how to learn faster.

Here is how CAUKER works, broken down into simple concepts:

1. The "Recipe Book" (Gaussian Processes)

Imagine the Mud Factory has a giant recipe book for making footprints.

• The Ingredients: It has recipes for "Trend" (a straight line), "Seasonality" (a wavy pattern like a heartbeat), and "Noise" (random squiggles).
• The Magic: Instead of just picking one recipe, the factory randomly mixes and matches them. It might take a "wavy" recipe and multiply it by a "spiky" recipe. This creates footprints that look complex and realistic, just like real animal tracks.

2. The "Family Tree" (Causal Models)

This is the secret sauce. In the real world, footprints aren't just random; they are connected. A lion's track might lead to a river, which leads to a watering hole.

• CAUKER doesn't just print random footprints. It builds a family tree for them.
• It starts with a "parent" footprint (generated by the recipe book).
• Then, it creates "child" footprints by applying rules (like "make it faster" or "make it wobbly").
• This ensures the fake data has logic and cause-and-effect, just like the real world. The robot learns not just what a footprint looks like, but why it looks that way.

3. The "Training Gym" (Pre-training)

The researchers used this factory to generate 10 million fake footprints. They fed this massive amount of data to two different robot brains (called Mantis and MOMENT) to train them.

• The Result: These robots, trained only on fake data, became experts. When they were tested on real footprints they had never seen before, they performed just as well as robots trained on millions of real footprints.
• The Surprise: Usually, robots trained on fake data are dumb. But because CAUKER's fake data was so diverse and logically structured, the robots learned the rules of footprints so well that they could handle anything.

4. The "Scaling Law" (Bigger is Better)

One of the coolest findings is about scaling.

• Real Data Problem: If you give a robot more real footprints, its performance often hits a wall. It gets stuck because the real data is messy, repetitive, or missing certain types of animals. It's like trying to learn a language by only reading one newspaper; you get stuck.
• CAUKER Success: With CAUKER, the more fake footprints you generate, the smarter the robot gets. It's a straight line up! If you double the fake data, the robot gets significantly better. This proves that quality and diversity of data matter more than just the source.

Why This Matters

Think of CAUKER as a flight simulator for time series data.

• Before, pilots (AI models) had to fly real planes (real data) to learn, which was dangerous and limited by weather.
• Now, CAUKER is a simulator that can generate infinite, perfect storm scenarios, engine failures, and clear skies instantly.
• The pilot trained in this simulator is so well-prepared that when they finally get into a real plane, they handle it perfectly, even if they've never seen that specific storm before.

In short: The paper shows that we don't need to wait years to collect massive amounts of real-world data. We can build a smart, logical "fake data factory" that trains AI models faster, cheaper, and often better than the real thing.

Technical Summary

Here is a detailed technical summary of the paper "CAUKER: Classification Time Series Foundation Models Can Be Pretrained on Synthetic Data."

1. Problem Statement

Time Series Foundation Models (TSFMs) have achieved remarkable zero-shot capabilities in forecasting and classification by pre-training on massive, curated collections of real-world data (often billions of timepoints). However, this approach faces significant challenges:

• Data Scarcity & Curation: Collecting diverse, high-quality real-world time series for classification is difficult and time-consuming.
• Scaling Irregularities: Real-world classification datasets (e.g., UCR, UEA) often exhibit heterogeneous, imbalanced, and small-scale distributions, leading to irregular or non-existent scaling laws when training larger models.
• Data Leakage Risks: Pre-training on real-world benchmarks often inadvertently includes training splits of the test sets, compromising the validity of zero-shot evaluation.
• Synthetic Data Limitations: Existing synthetic data generators are tailored for forecasting (focusing on smooth extrapolation) or tabular data (ignoring temporal dynamics), making them suboptimal for time series classification, which requires distinct, separable clusters and realistic temporal dependencies.

The core question addressed is: Can TSFMs for classification be effectively pre-trained solely on synthetic data that is cheaper to generate, scalable, and free from data leakage?

2. Methodology: The CAUKER Framework

The authors propose CAUKER (Causal-Kernel generation), a novel synthetic data generation pipeline designed specifically for time series classification. It unifies two distinct concepts: Gaussian Process (GP) Kernel Composition and Structural Causal Models (SCM).

Key Components:

1. Function Banks:

   • Kernel Bank ($\mathcal{K}$): Contains diverse kernels (e.g., RBF, ExpSineSquared, DotProduct) to model periodicity, trends, and smoothness.
   • Mean Bank ($\mathcal{M}$): Includes linear, exponential, and anomaly mean functions to introduce discriminative level shifts and non-stationarity (crucial for classification).
   • Activation Bank ($\mathcal{A}$): Contains non-linear functions (ReLU, Sigmoid, Sine, Modulo, Leaky ReLU) to introduce complex non-linearities.

2. Generation Pipeline (5 Steps):

   • Step 1 & 2 (Kernel Composition): Randomly sample kernels from $\mathcal{K}$ and combine them using binary operations (+, $\times$) to create composite kernels.
   • Step 3 (Root Node Generation): Sample mean functions from $\mathcal{M}$ and combine them with the composite kernels to define GP priors. Time series are sampled from these GPs to serve as root nodes (in-degree zero) in a causal graph. Crucially, unlike forecasting generators, CAUKER retains non-zero means to act as discriminative cues.
   • Step 4 (Activation Sampling): Sample activation functions from $\mathcal{A}$.
   • Step 5 (Causal Graph Propagation): A Directed Acyclic Graph (DAG) is generated. Non-root nodes are computed by aggregating incoming edges (linear transformation) and applying the assigned activation function. This propagates the root signals through the graph, creating complex, causally coherent multivariate (or univariate) time series.

3. Design Philosophy:

   • Temporal Realism: The GP component ensures realistic trends, seasonality, and smoothness.
   • Causal Coherence: The SCM component ensures that samples within the same "cluster" share underlying causal structures, creating meaningful separability for classification tasks.
   • Diversity: Random sampling of kernels, means, and graph structures ensures a vast diversity of patterns.

3. Key Contributions

1. Novel Synthetic Generator: Introduction of CAUKER, the first pipeline specifically designed to generate synthetic data for time series classification by combining GP kernels (for temporal motifs) and SCMs (for causal structure).
2. Sample-Efficient Pre-training: Demonstration that TSFMs (Mantis and MOMENT) pre-trained solely on CAUKER data can match or exceed the performance of models pre-trained on massive real-world corpora (e.g., 1.89M real series vs. 100K synthetic series for Mantis).
3. Discovery of Scaling Laws:
   • Data Scaling: CAUKER-generated datasets exhibit clear, monotonic scaling laws (accuracy increases with dataset size from 10K to 10M). In contrast, real-world datasets (UEA) show irregular or flat scaling due to lack of diversity.
   • Model Scaling: Models pre-trained on CAUKER show consistent performance gains as model capacity increases (1M to 783M parameters), whereas models pre-trained on real data often plateau or degrade.
4. Zero-Shot Superiority: CAUKER-pretrained models achieve state-of-the-art zero-shot performance on the UCR benchmark, outperforming models trained on real-world data that suffer from in-distribution bias.
5. Generalizability: The method transfers effectively to forecasting (Chronos models) and irregular clinical time series, suggesting the learned representations are robust and generalizable.

4. Experimental Results

The authors evaluated CAUKER using two state-of-the-art TSFMs: Mantis (contrastive, encoder-only) and MOMENT (masked, encoder-decoder).

• Comparison with Alternatives: CAUKER outperformed other synthetic generators (FPFN, KernelSynth, Mean+KernelSynth, and TabPFN's SCM) on the UCR benchmark.
  • Mantis: 78.31% (CAUKER) vs. 77.70% (KernelSynth).
  • MOMENT: 74.24% (CAUKER) vs. 72.56% (Mean+KernelSynth).
  • Insight: Purely kernel-based methods lack discriminative clusters; purely SCM-based methods lack temporal motifs. CAUKER bridges this gap.
• Scaling Laws:
  • Data: Accuracy on UCR steadily increased with CAUKER data size (10K $\to$ 10M). Real-world UEA data showed no consistent trend.
  • Model: Larger models trained on CAUKER consistently improved. Larger models trained on UEA often degraded or plateaued.
• Sample Efficiency:
  • Mantis: Pre-trained on 100K CAUKER samples achieved 78.55% accuracy, nearly matching the original model trained on 1.89M real samples (78.66%).
  • MOMENT: Pre-trained on 10M CAUKER samples achieved 77.49%, close to the original 13M real-sample baseline (78.85%).
• Qualitative Analysis:
  • Embeddings: PCA and UMAP visualizations showed CAUKER embeddings cover a broader, more diverse region of the embedding space than real-world data.
  • Attention: Attention rollout analysis revealed CAUKER-trained models focus more sharply on discriminative temporal segments compared to real-data trained models.
  • Non-linearity: Models trained on larger CAUKER datasets exhibited increased non-linearity in their internal representations, indicating better feature learning.

5. Significance and Impact

• Paradigm Shift: The paper challenges the dogma that foundation models require massive, curated real-world datasets. It proves that principled synthetic data design can be a more effective and scalable alternative for classification tasks.
• Cost & Accessibility: Synthetic data generation is computationally cheaper and faster than data collection/curation, democratizing access to high-quality pre-training for TSFMs.
• Evaluation Integrity: By using purely synthetic pre-training, the risk of data leakage in zero-shot evaluation is eliminated, leading to more trustworthy benchmarks.
• Future Directions: The findings suggest that the "quality" and "structure" of pre-training data are more critical than sheer volume. This encourages the community to focus on designing better synthetic generators and analyzing data distributions rather than just scaling up data collection.

In conclusion, CAUKER demonstrates that by synthesizing data with realistic temporal dynamics and causal structures, we can train robust, scalable, and generalizable Time Series Foundation Models without relying on expensive or leak-prone real-world datasets.