🤖 machine learning

TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Models

This paper introduces TSFMAudit, the first method designed to detect pretraining data contamination in Time Series Foundation Models by leveraging the intuition that contaminated datasets exhibit unusually efficient adaptation dynamics during fine-tuning.

Original authors: Hongkai Li, Shifeng Xie, Lefei Shen, Zhuo Li, Mouxiang Chen, Xiaobin Zhang, Han Fu, Jianling Sun, Xiaoxue Ren, Chenghao Liu

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongkai Li, Shifeng Xie, Lefei Shen, Zhuo Li, Mouxiang Chen, Xiaobin Zhang, Han Fu, Jianling Sun, Xiaoxue Ren, Chenghao Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a chef to cook a complex dish for a big dinner party. You want to know if this chef is truly a genius who can cook anything from scratch, or if they just memorized the recipe for the specific dish you are about to serve them.

This is the problem TSFMAudit solves, but instead of chefs and recipes, it's about Time Series Foundation Models (super-smart AI that predicts things like stock prices, weather, or traffic) and data contamination.

Here is the breakdown of the paper in simple terms:

The Problem: The "Leaked Test"

In the world of AI, researchers train models on massive amounts of data (the "pretraining corpus") and then test them on specific datasets to see how good they are.

The problem is that sometimes, the data used for the test accidentally ends up in the training pile.

The Analogy: Imagine a student studying for a math exam. If the teacher accidentally puts the exact test questions in the study guide, the student will get a perfect score. But that doesn't mean they are a math genius; it just means they memorized the answers.
Why it's hard with Time Series: In text (like books), you can easily spot if a sentence was copied. But time series data (like a graph of temperature over a year) is continuous. The same pattern can look different if you change the units (Celsius to Fahrenheit) or the scale. So, it's very hard to tell if the AI "saw" the data before or if it's just naturally good at predicting that specific pattern.

The Solution: The "Probe" Test

The authors created a tool called TSFMAudit. Instead of just looking at the final score (which can be faked by memorization), they watch how the AI learns when you give it a tiny bit of new training.

Think of it like a fitness test:

The Setup: You have a "Candidate" AI (the one you are auditing) and a group of "Reference" AIs (fresh models that haven't seen the data yet).
The Probe: You give them all a tiny, short workout (a few minutes of fine-tuning) on the specific dataset you are suspicious of.
The Observation:
- The Clean AI: If the AI has never seen this data, it has to work hard. It sweats a lot (its internal settings change a lot) and its performance improves slowly.
- The Contaminated AI: If the AI already saw this data during its big training phase, it recognizes the pattern immediately. It barely breaks a sweat (its internal settings barely move) and its performance jumps up instantly.

The Key Insight: Contamination looks like unusually efficient adaptation. The "cheater" learns faster with less effort.

How They Make It Fair (The "Reference Suite")

You might say, "But what if the data is just really easy to predict? Then even a clean AI would learn fast."

To fix this, the authors use a Reference Suite. They run the same test on four other models that are similar in size but definitely haven't seen the data.

The Analogy: Imagine you are judging a runner. If they run a 10-second 100m, that's amazing. But if the track is a downhill slope, maybe they aren't that fast. So, you have other runners run on the same track. If your runner is significantly faster than everyone else on the same easy track, you know they have an unfair advantage (like a hidden motor).
TSFMAudit compares the Candidate's speed against the Reference models. If the Candidate is way too efficient compared to the References, it's flagged as contaminated.

What They Found

The team tested this on 6 different Time Series AI models and 187 different datasets.

Old Methods Failed: They tried old tricks used for text models (like checking if the loss score is too low), but those didn't work well for time series because some data is just naturally easy to predict.
TSFMAudit Worked: By watching the "sweat" (how much the model changed) and the "speed" (how fast the score improved) during the probe, they could spot the cheaters much better than any other method.
Real-World Check: They tested their tool on a brand-new benchmark called TIME. Since this benchmark was built after the models were trained, it should be clean. TSFMAudit confirmed that these models hadn't seen the TIME data, proving the tool works in real life.

Summary

TSFMAudit is a "lie detector" for time-predicting AI. It doesn't just ask, "Did you get a good score?" It asks, "How hard did you have to work to get that score?" If the AI learns too easily compared to its peers, it's likely because it already knew the answers. This helps researchers ensure that when they say an AI is smart, it's actually smart, not just a cheater who memorized the test.

Technical Summary: TSFMAudit – Data Contamination Auditing in Forecasting Time Series Foundation Models

1. Problem Formulation

Time Series Foundation Models (TSFMs) are increasingly pretrained on large, heterogeneous corpora derived from diverse real-world and synthetic sources. Concurrently, widely used evaluation benchmarks (e.g., GIFT-Eval, TIME) are often assembled from the same public sources. This overlap creates a significant risk of pretraining contamination, where evaluation data has been exposed during the model's pretraining phase. Such exposure can artificially inflate downstream performance estimates, leading to overly optimistic assessments of a model's generalization capabilities.

Auditing this contamination in time series is uniquely challenging compared to Large Language Models (LLMs) due to the nature of the data:

Continuity and Heterogeneity: Time series are continuous values, making exact string or token matching (used in LLM overlap detection) inapplicable.
Preprocessing Sensitivity: The same underlying signal can reappear after rescaling, renaming, rewindowing, or other transformations, rendering simple duplicate detection brittle.
Lack of Documentation: Corpus-level documentation is often incomplete, making it difficult to verify exposure via metadata alone.
Unreliability of Static Metrics: Static loss or perplexity scores are unreliable indicators because smooth seasonal trends or easy datasets can yield low loss even for models that have never seen the data.

The paper formalizes the contamination auditing objective: given a candidate TSFM $f_\theta$ and an audited dataset $D$ , the goal is to estimate a score $S(f_\theta, D)$ that quantifies the risk of pretraining contamination, distinguishing between "clean" and "contaminated" pairs without access to the full pretraining corpus.

2. Methodology: TSFMAudit

The authors propose TSFMAudit, a framework based on probe adaptation dynamics. The core intuition is that contamination manifests as unusually efficient adaptation during a fine-tuning probe.

Key Observations:

Efficient Adaptation: If a model has already encoded predictive structures specific to a dataset during pretraining, it requires less adaptation when subsequently fine-tuned on that same dataset. This results in faster loss reduction with minimal movement of the backbone parameters.
Dataset Difficulty Bias: Fast adaptation is not unique to contaminated models; clean models may also adapt quickly if the dataset is intrinsically easy. To mitigate this, the method compares the candidate against a Reference Suite of models with comparable architecture but no exposure to the target dataset.

The TSFMAudit Pipeline:

Probing Phase: Both the candidate model and a suite of reference models ( $\mathcal{R}$ ) are fine-tuned on the audited dataset $D$ for a fixed number of epochs ( $T_{probe}$ ).
Trace Extraction: For each epoch $t$ $t$ , the system records:
- $\ell_t$ : The forecasting loss.
- $w_t$ : The cumulative $\ell_2$ displacement of backbone parameters from their initial state.
- Derived metrics: Relative loss drop ( $d_t$ ) and adaptation efficiency ( $a_t = d_t / w_t$ ).
Debiasing via Reference Comparison: To remove dataset-specific difficulty bias, the candidate's trace features are compared element-wise against each reference model's trace. This generates difference ( $\Delta$ ) and ratio ( $\rho$ ) features.
Scoring and Calibration: A lightweight scorer (logistic regression) is trained on an audit training set using these debiased features to predict contamination probability.
- Thresholding: The system adopts a conservative "do-not-accuse-the-innocent" principle (FP-0 calibration). The decision threshold $\tau$ is set to the maximum score observed on known clean validation datasets, ensuring no clean dataset is falsely flagged as contaminated.

3. Key Contributions

First Systematic Study: To the best of the authors' knowledge, this is the first work to formally study and propose a method for pretraining contamination auditing specifically for TSFMs.
Novel Methodology: The introduction of TSFMAudit, which leverages probe-time adaptation dynamics (loss reduction vs. parameter movement) rather than static loss or memorization extraction.
Reference-Based Debiasing: A mechanism to control for dataset difficulty by comparing candidate adaptation efficiency against a suite of reference models trained from scratch or on unrelated data.
Comprehensive Evaluation: The method is evaluated on 6 TSFMs (Chronos, TiRex, Moirai1/2, TimesFM2.0, Kairos) and 187 datasets (combining GIFT-Eval and TIME benchmarks), comparing against 10 baselines adapted from LLM literature.

4. Experimental Results

The experiments address four primary questions regarding the method's efficacy:

Detection Performance (Q1): TSFMAudit significantly outperforms static-loss baselines (Raw Loss, Loss Drop Rate), LiRA-inspired ratios, and frequency-domain baselines. While static metrics yield near-zero Matthews Correlation Coefficients (MCC), TSFMAudit achieves an MCC of 0.125 (All Ref configuration) and a Macro-F1 of 0.521. The authors note that absolute MCC values remain modest, reflecting the inherent difficulty of the problem under conservative FP-0 calibration and noisy proxy labels.
Debiasing Effect (Q2): Reference-based debiasing is crucial. All reference-based variants outperform the candidate-only dynamics baseline. The "All Ref" configuration (using all four reference models: ScratchCNN, ScratchTransformer, Stat, VisionTS) yields the best aggregate performance, confirming that relative adaptation signals are more informative than absolute ones.
Feature Utility (Q3): Feature ablation shows that no single feature is sufficient. Adaptation efficiency ( $a_t$ ) is the strongest individual feature, supporting the hypothesis that contamination is characterized by high efficiency (fast loss drop, low parameter movement). However, combining all dynamics features (loss, displacement, efficiency) yields the best results.
Practical Deployment (Q4): The method was tested on the TIME Benchmark, a fresh external benchmark with no contamination labels. Using a fixed auditor trained on previous data, TSFMAudit correctly identified the TIME datasets as "clean" (non-leaked) with high confidence (pooled non-leakage rate > 95%), demonstrating its utility in real-world deployment scenarios without retraining.

5. Significance and Claims

The paper claims that TSFMAudit establishes probe-time dynamics as a promising foundation for reliable contamination auditing in time series.

Paradigm Shift: It moves away from static, pre-probe metrics (which are unreliable in time series) toward dynamic, behavioral analysis during fine-tuning.
Feasibility: It demonstrates that contamination auditing is feasible even without access to the pretraining corpus or the ability to reproduce pretraining at scale, provided one can probe the model's adaptation behavior.
Limitations: The authors remain modest about the results, acknowledging that the detection problem is challenging. The modest MCC scores indicate that while the method is superior to baselines, it is not a perfect detector, particularly under strict false-positive constraints and when dealing with proxy labels that may not capture all forms of leakage (e.g., transformed duplicates).

In conclusion, TSFMAudit provides the first dedicated framework for auditing TSFMs, offering a robust, dynamic approach to identifying data contamination that accounts for the continuous and heterogeneous nature of time series data.