🤖 machine learning

Statistical comparisons of time-series feature sets on classification tasks

This study benchmarks six open-source time-series feature sets against three baselines across 124 classification problems, revealing that while most sets perform similarly overall, the comprehensive tsfresh library yields the highest win rate, yet specific problem characteristics often favor simpler baseline features.

Original authors: Trent Henderson, Ben D. Fulcher

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Trent Henderson, Ben D. Fulcher

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of fingerprints or footprints, your clues are lines on a graph that change over time. These lines are called time series, and they show up everywhere: the rhythm of a heartbeat, the fluctuation of stock prices, the movement of a robot's arm, or the temperature of a star. The goal is to look at these squiggly lines and figure out what "category" they belong to—like telling if a heartbeat is normal or if a robot is walking or falling.

To solve these puzzles, scientists use a toolkit of features. Think of a feature as a specific question you ask the data. "How fast is it moving?" "Is it getting hotter?" "Does it have a repeating pattern?" Different toolkits ask different questions. Some toolkits ask just a few simple questions, while others ask hundreds of complex ones. For a long time, researchers had to guess which toolkit was the best for the job. They didn't know if the giant, complicated toolkit was actually better than the small, simple one, or if they were just asking the same questions in different ways. This paper steps into that detective room to see which toolkit actually solves the most mysteries.

The Great Feature Set Showdown

In this study, researchers Trent Henderson and Ben Fulcher decided to put six popular, free-to-use software toolkits (called "feature sets") to the test. They also included three very simple "baseline" toolkits to see if the fancy ones were even necessary. They didn't just look at one or two problems; they threw these toolkits into 124 different time-series classification challenges, ranging from identifying different types of coffee beans to detecting heart conditions from ECG signals.

The researchers wanted to know: Do the big, complex toolkits actually win more often? Or do the small, simple ones do just as well? And when does a specific toolkit fail spectacularly?

The "Tie" That Won the Day

The most surprising finding was that, for the vast majority of cases, it didn't really matter which toolkit you used.

Imagine a race with six runners. You might expect the runner with the most expensive shoes (the biggest toolkit) to win every time. But in this race, 85.3% of the time, the runners finished in a tie. Whether the toolkit had 22 features or 783 features, they usually got the same score. The researchers found that most of these toolkits were asking very similar questions about the data, just in slightly different languages. Because they were so similar, they usually ended up with the same answer.

The Giant Who Sometimes Wins

Despite the frequent ties, one toolkit stood out as the overall champion: tsfresh. This toolkit is the "giant" of the group, containing 783 features. It won 29.03% of the head-to-head comparisons against the other toolkits. It wasn't a landslide victory, but it was the most consistent winner.

However, the researchers found that size isn't everything. Sometimes, the smallest toolkits, like catch22 (only 22 features) or Kats (40 features), actually beat the giants on specific problems. For example:

Kats failed miserably on a problem called SyntheticControl because it lacked a specific feature to measure "linear trends" (straight lines going up or down). Without that one specific question, it couldn't solve the puzzle.
tsfresh absolutely crushed a problem called Beef. Why? Because it included a massive collection of Fourier coefficients (mathematical tools that break sounds or signals into their component frequencies). On this specific problem, the classes were distinguished by subtle frequency differences that the other toolkits missed because they summarized the data too broadly.

The Power of Simplicity

Perhaps the most playful twist in the story is that sometimes, you don't need a toolkit at all. The researchers tested three "baselines"—super simple sets of features that just looked at the shape of the data (quantiles) or its frequency (Fourier coefficients).

On several problems, these simple baselines performed just as well as, or even better than, the complex toolkits. For instance, on the PigArtPressure problem, a simple list of 101 quantiles (which just describes the shape of the data distribution) was enough to solve the mystery perfectly. This suggests that for many problems, you don't need a super-computer to analyze the data; you just need to understand the basic shape and rhythm of the signal.

What This Means for You

The authors conclude that there is no single "best" toolkit for every job. While the giant toolkit (tsfresh) is the safest bet if you want the highest chance of winning overall, it's not a magic bullet.

Don't just pick the biggest tool: Sometimes a small, specific tool is better.
Don't ignore the basics: Simple math can often solve complex problems just as well as fancy algorithms.
Know your problem: If you know your data has a specific quirk (like a specific frequency or a straight-line trend), you need a toolkit that asks about that specific thing.

The study suggests that instead of blindly picking a complex library, scientists should look at the specific problem they are trying to solve. If a simple baseline works, use it. If a specific feature is missing from your toolkit, that might be why you're losing the race. It's a reminder that in the world of data, sometimes the simplest questions yield the best answers.

Technical Summary: Statistical Comparisons of Time-Series Feature Sets on Classification Tasks

Problem Statement
In recent years, numerous open-source software libraries have been developed to extract sets of features from univariate time-series data. These feature sets vary significantly in their composition, size, and the disciplinary perspectives they employ to quantify temporal structure. Despite the availability of these tools, the relative strengths and weaknesses of different feature sets on time-series classification (TSC) problems remain largely unexplored. Existing benchmarking practices often rely on aggregated rank-based metrics, which the authors argue are too coarse to discern when and why a specific feature set performs well or poorly on a given problem. Furthermore, it is unclear whether the specific differences in feature composition translate into substantial performance gains or if simple baselines might suffice for many tasks.

Methodology
The study benchmarks six prominent open-source feature sets against three simple baseline sets across 124 univariate time-series classification problems from the UEA/UCR repository.

Feature Sets Evaluated:
- Open-source libraries: catch22 (22 features), tsfeatures (62 features), feasts (43 features), tsfresh (783 features), TSFEL (156 features), and Kats (40 features).
- Baseline sets: (i) 200 Fast Fourier Transform (FFT) coefficients (squared magnitude and angle); (ii) 101 quantiles (Q0.0–Q1.0); and (iii) a union of both (301 features).
Data Preprocessing: All time series were z-scored prior to feature extraction to standardize comparisons and prevent classifiers from relying on trivial distributional properties (mean/variance) rather than temporal structure.
Classification Approach: A ridge-regularized multinomial (or binomial) logistic regression classifier was trained on the extracted feature matrices. To ensure robustness, the study utilized 30 train-test resamples per problem. For problems with sufficient samples, 10-fold cross-validation was used to optimize the regularization penalty; otherwise, a fixed penalty ( $C=1.0$ ) was applied.
Statistical Benchmarking:
- Normalized Performance Score (NPS): To address the varying difficulty of problems, the authors introduced a problem-level z-score normalization. $NPS_{ij}$ represents the performance of feature set $j$ on problem $i$ relative to the mean and standard deviation of all feature sets on that specific problem. This allows for direct comparison of performance magnitudes across different problems.
- Pairwise Statistical Comparisons: Instead of simple rank aggregation, the authors employed correlated t-tests on the distribution of resampled classification accuracies. This accounts for the non-independence of resamples. A feature set is declared a "winner" against another if the p-value is $< 0.05$ and its mean accuracy is higher; otherwise, the result is recorded as a tie.

Key Results

Overall Similarity: Despite dramatic differences in size (from 22 to 783 features) and composition, the feature sets performed relatively similarly overall. 85.3% of all pairwise comparisons resulted in statistical ties.
Top Performer: The largest feature set, tsfresh, exhibited the strongest overall performance, winning 29.03% of all pairwise comparisons against other sets.
Small Set Success: Smaller sets like catch22 and Kats achieved performance significantly above the mean (NPS > 1) on specific problems (e.g., FreezerRegularTrain for catch22 and Yoga for Kats), demonstrating that unique feature compositions can be highly effective for specific problem types.
Baseline Competitiveness: Simple baselines were sufficient for strong performance on several problems.
- The quantiles baseline outperformed all dedicated feature sets on four problems (e.g., PigArtPressure), suggesting that distributional shape alone was sufficient to distinguish classes.
- The FFT baseline (and the union set) outperformed others on four problems (e.g., Beef, ChlorineConcentration), indicating that frequency-domain characteristics were the primary discriminators.
Feature Composition Impact: The study identified specific cases where feature composition drove performance. For instance, Kats underperformed on SyntheticControl because it lacked a feature sensitive to linear trend strength, a key differentiator for that problem. Conversely, tsfresh's superior performance on Beef was driven by its inclusion of a comprehensive set of raw Fourier coefficients, whereas other sets summarized spectral properties at a coarser level.

Significance and Claims
The paper claims that its primary contribution is a shift in benchmarking methodology from rank-based aggregation to magnitude-sensitive, problem-level analysis. The authors argue that:

Rank-based methods are insufficient: Aggregating performance into average ranks imposes a false hierarchy on statistically similar results and masks the specific conditions under which a feature set excels or fails.
Feature composition matters: While common statistical properties drive broadly similar performance, the specific "make-up" of a feature set is the key driver of above-average or below-average performance on specific problems.
The value of baselines: The results highlight the necessity of comparing complex feature sets against simple, interpretable baselines (like quantiles and FFTs) to justify the use of more computationally intensive methods.
Future Directions: The authors suggest that future work could focus on constructing reduced, bespoke feature sets derived from the union of existing libraries, optimized for specific problem types to balance performance, redundancy, and computation time. They also note the potential for applying these comparative methods to other tasks like forecasting and extrinsic regression.