🤖 machine learning

TSDS-Toolbox: A Toolbox for Measuring Time-Series Dataset Similarity

This paper introduces TSDS-Toolbox, a unified and extensible framework designed to address the fragmentation in existing benchmarks by enabling systematic, reproducible, and consistent evaluation of time-series dataset similarity methods for tasks like fine-tuning foundation models.

Original authors: Yen-Ku Liu, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yen-Ku Liu, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to invent a new recipe. You have a target dish in mind—maybe a spicy noodle soup—but you don't know which existing recipe book will help you learn the best. Do you start with a French cookbook, a Thai street-food guide, or a classic Italian manual? In the world of artificial intelligence, specifically with "time-series" data (which is just a fancy way of saying data that changes over time, like stock prices, heartbeats, or weather patterns), AI models face the same problem. They need to learn from old data to predict the future or spot weird patterns. But not all old data is created equal. Some datasets are like cousins to your target dish; they share the same flavor profile. Others are like total strangers. Figuring out which datasets are "similar" enough to be useful is a huge challenge. Right now, scientists have many different ways to measure this similarity, but they are all using different rulers, different scales, and different kitchens, making it impossible to compare them fairly.

This is where the TSDS-Toolbox comes in. Think of it as a massive, standardized "Taste-Test Kitchen" built by researchers Yen-Ku Liu, Hongjie Chen, Ryan A. Rossi, and Franck Dernoncourt. Before this tool existed, if you wanted to know if Dataset A was more similar to Dataset B than Dataset C, you might have to write your own code, clean your own data, and run your own experiments, only to find out your results couldn't be compared to anyone else's. The authors built a unified framework that acts like a giant, automated judge. It takes in different "similarity methods" (the different ways to measure how alike two groups of time-series data are) and tests them all under the exact same conditions. They didn't just build a ruler; they built a whole lab to test if the ruler actually helps you cook better.

The team put their toolbox to the test using 25 different real-world datasets, ranging from traffic patterns and weather reports to electricity usage and tourism numbers. They asked a simple but tricky question: "Does knowing that two datasets are 'similar' actually help an AI model perform better on a new task?" They tested this by seeing if the similarity scores could predict how well an AI would do at forecasting (predicting the future) or classification (sorting things into categories).

Here is what they found, and it's a bit of a plot twist. They discovered that there is no single "magic ruler" that works best for every situation. It's like asking if a tape measure, a laser distance finder, or a pedometer is the best tool for measuring distance. The answer depends entirely on what you are measuring and why.

For some tasks, like predicting the future with a specific type of AI model called Time-MoE, a method called Match-and-Deform (which is like matching two dance routines even if they are slightly out of sync) turned out to be the best predictor of success.
For other tasks, a method called Wasserstein Distance (which measures the "cost" of moving data from one shape to another) performed very well.
Interestingly, some methods that seemed great on paper didn't help the AI perform better in practice.

The researchers also tested two different ways of simplifying the data before measuring it: DBA (which averages out the data like finding the "average" dance move) and PCA (which finds the most important "directions" in the data). They found that DBA was generally better at helping the AI learn, especially for classification tasks, but PCA held its own in specific forecasting scenarios.

The most important takeaway from this study is that you can't just pick one similarity method and assume it's the best for everything. The paper suggests that the "best" way to measure similarity depends entirely on what you are trying to do with the data. If you are trying to predict the weather, one tool might be your best friend; if you are trying to detect a heart attack, a completely different tool might be the hero.

By providing this open-source toolbox, the authors have given the scientific community a way to stop guessing and start testing. Instead of arguing about which ruler is best in theory, researchers can now run their own experiments in this shared kitchen to see exactly which tool helps their specific AI model cook up the best results. It's a step toward making AI smarter by helping it choose the right ingredients from the start.

Technical Summary: TSDS-Toolbox

Problem Statement
The rapid advancement of AI, particularly foundation models, has increased the reliance on time-series dataset similarity for tasks such as source dataset selection, fine-tuning, and transfer learning. While quantifying the similarity between time-series datasets is critical for downstream utility (e.g., determining if a source dataset provides useful signals for a target domain), existing research lacks a unified framework. Current implementations are fragmented, often focusing on individual time-series comparison rather than dataset-level similarity, and require researchers to manually integrate metrics, standardize formats, and build evaluation pipelines from scratch for every study. This inconsistency hinders fair benchmarking and reproducible research.

Methodology: The TSDS-Toolbox Framework
To address these limitations, the authors propose the Time-Series Dataset Similarity Toolbox (TSDS-Toolbox), a unified, configuration-driven, and extensible framework. The system is organized into four core layers:

Data Layer: Handles heterogeneous input formats (GluonTS, CSV, NPY, Pandas DataFrames) and converts them into a unified representation. It applies consistent preprocessing, including sampling, fixed-length window extraction, reshaping, train-validation splitting, and normalization.
Similarity Layer: Computes pairwise distance matrices using two distinct approaches:
- Dataset-Level Metrics: Directly compute dissimilarity between two datasets without reduction. Supported methods include Wasserstein Distance (WSD), Maximum Mean Discrepancy (MMD), Optimal Transport (OT), and Match-and-Deform (MAD). These often treat datasets as empirical distributions.
- Series-Level Metrics: First reduce each dataset to a representative sequence using a Reducer (DTW Barycenter Averaging [DBA] or Principal Component Analysis [PCA]), then compute distances between these representatives. Supported distances include Dynamic Time Warping (DTW), Euclidean Distance (ED), and Longest Common Subsequence (LCSS).
Evaluation Layer: Assesses the practical utility of similarity scores by correlating them with downstream task performance. It supports two settings:
- Out-of-Distribution (OOD) Evaluation: Tests if a model fine-tuned on a source dataset performs well on a target dataset.
- Transfer-Learning (TL) Evaluation: Tests adaptation scenarios where a model is fine-tuned on a source, further adapted on a small target split, and evaluated on a target inference split.
- Tasks: The toolbox evaluates both Classification (using a one-class $k$ -NN membership criterion) and Forecasting (using foundation models like Lag-Llama and Time-MoE).
Analysis Layer: Visualizes dataset relationships via heatmaps and network graphs. It quantitatively measures the alignment between similarity scores and downstream task errors using Pearson correlation.

Key Contributions
The paper outlines three primary contributions:

Unified Benchmarking Interface: A framework enabling fair, systematic comparison of time-series dataset similarity methods under consistent experimental settings.
Standardized Downstream Evaluation: Pipelines that assess the practical utility of dataset similarity for real-world time-series tasks (classification and forecasting).
Modular Design: A configuration-driven architecture that supports reproducibility and allows users to easily extend the toolbox with new datasets, similarity metrics, reducers, and evaluation tasks.

Experimental Results
The authors validated TSDS-Toolbox using 25 GluonTS datasets spanning traffic, weather, electricity, exchange rates, tourism, and public health. Key findings include:

Task Dependency: No single similarity metric consistently achieves the highest correlation with downstream performance across all tasks. For instance, in forecasting out-of-distribution (OOD) tasks with Time-MoE, Match-and-Deform (MAD) showed the strongest correlation, followed closely by Wasserstein Distance (WSD).
Reducer Effectiveness: The choice of reducer significantly impacts results. DTW Barycenter Averaging (DBA) generally achieved stronger correlations than Principal Component Analysis (PCA), particularly for classification and Time-MoE forecasting. However, PCA remained competitive for Lag-Llama OOD forecasting.
Metric Performance: Series-level metrics using DBA and Euclidean Distance remained competitive, while LCSS and MMD showed weaker alignment with downstream performance in many settings.
Visualization: The toolbox successfully generated heatmaps and network graphs that revealed how different metrics induce distinct dataset relationship structures.

Significance and Claims
The paper claims that TSDS-Toolbox addresses the limitations of existing fragmented tools by providing a unified framework for benchmarking dataset similarity. Its significance lies in:

Enabling Reproducibility: By standardizing the integration of metrics and evaluation pipelines, it removes the need for researchers to rebuild infrastructure for every study.
Informing Source Selection: It provides empirical evidence that dataset similarity is task-dependent, suggesting that researchers should not rely on a single metric but rather use systematic evaluation to select appropriate source datasets for fine-tuning and transfer learning.
Foundation for Future Research: The modular design offers an extensible foundation for future investigations into time-series dataset similarity, supporting the development of better foundation models and adaptation strategies.

The authors conclude that systematic evaluation, facilitated by TSDS-Toolbox, is necessary to move beyond ad-hoc metric selection and toward practical, data-driven source dataset selection.

Technical Summary: TSDS-Toolbox

More like this