Unsupervised Feature Based Algorithms for Time Series Extrinsic Regression

Technical Summary: Unsupervised Feature Based Algorithms for Time Series Extrinsic Regression

Problem Definition
Time Series Extrinsic Regression (TSER) involves constructing a predictive model to estimate a continuous response variable based on a set of training time series, where the response is not directly derived from the series itself (unlike forecasting). While Time Series Classification (TSC) and forecasting are well-studied, TSER has received less attention. A previous benchmark study [5] established an archive of 19 problems and evaluated 13 algorithms, finding that deep learning (InceptionTime) and kernel-based methods (ROCKET) performed well, though no single algorithm significantly outperformed the others. The authors argue that the field requires further research to improve model accuracy and address the lack of progress since the initial benchmark.

Methodology
The study expands the TSER landscape through three primary methodological steps:

1. Archive Expansion: The authors increased the TSER benchmark archive from 19 to 63 datasets. This new collection includes 24 univariate and 20 multivariate datasets (in addition to the original 19), sourced from Kaggle competitions and applied research repositories. The datasets cover diverse domains such as economic analysis, energy monitoring, environmental science, equipment health, and sentiment analysis. Data preprocessing involved linear interpolation for missing values and truncation for unequal-length series.
2. Algorithm Adaptation and Implementation: The study adapts state-of-the-art TSC algorithms for regression tasks. The core focus is on unsupervised feature-based approaches, which transform time series into feature vectors without using the target variable during the transformation phase.
   • FreshPRINCE: A pipeline combining the TSFresh feature extraction library (generating ~800 unsupervised features) with a Rotation Forest (RotF) regressor. The RotF was adapted for regression by replacing the C4.5 classifier with a CART regressor and averaging predictions.
   • DrCIF (Diverse Representation Canonical Interval Forest): An interval-based ensemble. It splits time series into three representations: the original series, first-order differences, and periodograms. It then selects random intervals within these representations and extracts summary features (using the Catch22 set plus seven standard statistics like mean and slope). An ensemble of CART regressors is trained on these concatenated features.
   • Other Baselines: The study evaluates 21 regressors in total, including standard ML models (Ridge, SVR, Random Forest, XGBoost), functional linear models (FPCR), distance-based methods (kNN with Euclidean and DTW), convolutional/kernel methods (ROCKET, MultiROCKET), and deep learning architectures (FCN, ResNet, InceptionTime single and ensemble, CNN).
3. Experimental Design: Experiments were conducted using 30 resamples (train/test splits) per dataset to ensure statistical robustness. Performance was measured using Root Mean Square Error (RMSE). Statistical significance was assessed using pairwise Wilcoxon signed-rank tests with Holm correction, visualized via Critical Difference (CD) diagrams.

Key Contributions

1. Dataset Expansion: The release of 44 new datasets, bringing the total TSER archive to 63 problems.
2. Algorithmic Innovation: The introduction of two new TSER algorithms, FreshPRINCE and DrCIF, adapted from successful TSC methodologies.
3. Comprehensive Benchmarking: A rigorous evaluation of 21 distinct regressors, including standard off-the-shelf models, functional analysis, and deep learning, providing a new baseline for the field.
4. Open Source: Provision of scikit-learn compatible implementations and detailed reproducibility guidelines.

Results

• Baseline Reproduction: On the original 19 datasets, the study reproduced the findings of [5], confirming that no single algorithm significantly dominated the others, though the ordering of algorithms shifted slightly with less conservative statistical corrections.
• Extended Archive Performance: On the full 63-dataset archive, standard regressors like Random Forest and Rotation Forest (RotF) outperformed many time-series-specific deep learning and kernel methods. Notably, RotF emerged as a top-performing standard algorithm.
• Superiority of New Proposals: The study demonstrates that FreshPRINCE and DrCIF significantly outperform all other 18 regressors tested, including the best-performing standard regressor (RotF) and the deep learning ensemble (InceptionTimeE).
  • DrCIF and FreshPRINCE achieved the lowest average RMSE ranks.
  • InceptionTimeE, while often top-ranked on specific problems, showed high variance and instability, frequently failing to find predictive relationships in datasets where other models succeeded.
  • Ablation Study: For FreshPRINCE, the study confirmed that the performance gain comes from the specific combination of the TSFresh transform and the RotF regressor; using TSFresh with other regressors (like Random Forest or XGBoost) did not yield the same results as the RotF pairing.
• Data Characteristics: Performance varied by data properties. DrCIF performed relatively better on short series and multivariate data with fewer channels, while FreshPRINCE excelled with larger training sets and datasets with more than 5 channels.

Significance and Claims
The paper claims that the field of TSER has been stagnant and that the proposed unsupervised feature-based approaches represent a significant leap forward. The authors assert that:

• No previously evaluated regressor can outperform a regression adaptation of a standard classifier (specifically Rotation Forest).
• FreshPRINCE and DrCIF are the only models that significantly outperform the standard Rotation Forest regressor, establishing a new state-of-the-art for TSER.
• The diversity in performance between the top algorithms suggests that heterogeneous ensembles could be a fruitful area for future research.
• The study provides a more robust and larger benchmark (63 datasets) that reveals limitations in previous conclusions drawn from smaller datasets.

The authors remain modest regarding limitations, noting that the data is not randomly sampled (potentially over-representing certain domains), no hyperparameter tuning was performed due to computational constraints, and complex diagnostic analyses (like residual analysis) were not conducted. However, they conclude that their work provides a significant contribution to advancing the TSER field by demonstrating that unsupervised feature extraction combined with robust ensemble regressors is currently the most effective approach.