🤖 machine learning

Revisiting WEASEL 2.0: Reproduction, Sensitivity, and an Adaptive Ensemble-Size Rule

This paper reproduces the WEASEL 2.0 time series classifier to validate its performance and identifies that its fixed ensemble-size rule is inefficient for long-series datasets, leading to the proposal of an adaptive rule that significantly reduces memory usage and training time with negligible impact on accuracy.

Original authors: Cian Higgins, Gerard Carrigan, Pinar Sungu Isiacik, Georgiana Ifrim

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cian Higgins, Gerard Carrigan, Pinar Sungu Isiacik, Georgiana Ifrim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, there is a constant effort to teach computers how to recognize patterns in sequences of numbers that change over time. This field, known as time series classification, helps machines understand everything from the rhythm of a heartbeat in a medical monitor to the vibrations of a machine in a factory. One popular way to solve these puzzles involves a method called a dictionary approach. Imagine taking a long, continuous stream of data and slicing it into many small, overlapping pieces. The computer then translates each slice into a simple symbol, like a letter in a word, creating a collection of these symbols that acts as a dictionary for that specific data stream. By counting how often certain "words" appear, the computer can learn to distinguish one type of event from another. While this method has been around for years, it often struggles with two major problems: it can become incredibly slow and memory-hungry when the data gets long, and it can be overly sensitive to tiny, meaningless changes in the data. A newer version of this method, called WEASEL 2.0, was designed to fix these issues by using a smarter way to slice the data and a fixed, manageable size for its memory, promising to be both accurate and efficient.

Researchers at University College Dublin decided to put this promising new method to the test, not just to see if it worked, but to understand exactly how it worked and whether its settings were truly necessary. They began by running the software on a massive collection of 114 different data sets, which included everything from short bursts of sensor readings to long recordings of heartbeats. Their goal was to see if they could reproduce the original results that the method's creators had published. Using their own computers and a fresh copy of the code, they found that the numbers matched almost perfectly. The new version of the software achieved the same high levels of accuracy as the original, confirming that it is indeed a top-tier tool for sorting time-based data. This successful reproduction was the first step, proving that the foundation was solid and that the claims made by the original authors were trustworthy.

With the foundation verified, the team turned their attention to the specific rules the software uses to decide how much work to do. The original paper had suggested a few simple rules of thumb for setting the size of the data slices and the number of different configurations the computer should try, but these rules had never been rigorously tested to see if they were the best possible choices. The researchers systematically changed these settings to see what would happen. They tried swapping out the final decision-making part of the software for a different type, and they tested adding a weighting system that had worked well in older versions of similar software. In both cases, the changes made the software worse or no better than before. They also tested changing the maximum size of the data slices. They found that the original rule for slice size was robust; making the slices larger or smaller did not improve the results and sometimes made them worse. This confirmed that the original designers had chosen these specific settings wisely.

However, one rule stood out as potentially wasteful. The software had a rule for deciding how many different versions of itself to run at once, known as the ensemble size. The original rule suggested that for very long data streams, the computer should run a large number of these versions to ensure accuracy. The researchers discovered that for many of the long data streams, this rule was over-provisioned. The computer was doing far more work than it needed to, consuming hundreds of megabytes of memory and taking several extra seconds to finish, without actually gaining any meaningful improvement in accuracy. It was like sending a large crew to move a single piece of furniture when two people would have been enough.

To solve this, the team proposed a new, adaptive rule that looks at the length of the data and the number of categories it needs to sort into, rather than just the size of the data set. If the data is very long, the new rule allows for a larger number of versions to run, but if the data is short or the categories are simple, it drastically reduces the number of versions. When they tested this new approach on the fixed-length data sets, the results were striking. The software ran faster and used significantly less memory, with a median reduction of 37 megabytes in peak memory usage and a saving of 0.4 seconds in fitting time per run. Crucially, this efficiency came with almost no cost to accuracy; for the majority of data sets, the accuracy remained exactly the same. The researchers found that the savings were concentrated on the long data streams where the original rule had been most aggressive. By making the software smarter about when to work hard and when to work light, they managed to keep the high accuracy of the original method while making it much more efficient for the computers that run it.

Technical Summary: Revisiting WEASEL 2.0: Reproduction, Sensitivity, and an Adaptive Ensemble-Size Rule

Problem Statement
WEASEL 2.0 is a dictionary-based time series classifier (TSC) that utilizes dilated sliding windows and a randomized hyperparameter ensemble to produce a fixed-size, dense feature representation. While the original authors claimed it as the most accurate dictionary classifier, two critical hyperparameter choices—the maximum ensemble size and the maximum window size—were defined by simple thresholding rules without empirical justification. Furthermore, no independent reproduction of WEASEL 2.0 had been published to verify its reported accuracy or to examine the sensitivity of its design choices to perturbation. This work addresses the lack of independent verification and investigates whether these heuristic thresholds are robust or if they can be refined to reduce computational resource consumption without sacrificing accuracy.

Methodology
The study proceeds in three phases: independent reproduction, sensitivity analysis, and the proposal of an adaptive rule.

Reproduction: The authors independently reproduced WEASEL 2.0 on 114 UCR datasets using the aeon toolkit and scikit-learn's RidgeClassifierCV. They matched the original configuration (random seed 1379, 4 threads) and compared results against seven other classifiers (MultiRocket, MiniRocket, ROCKET, R-DST, Hydra, WEASEL 1.0, and cBOSS).
Sensitivity Analysis: Four design choices lacking empirical justification were tested on a representative subset of 22 UCR datasets:
- Downstream Classifier: Replacing the default Ridge classifier with Stochastic Gradient Descent (SGD).
- Feature Weighting: Applying TF-IDF and sublinear TF scaling to the raw word counts.
- Maximum Window Size ( $w_{max}$ ): Sweeping values from 12 to 200 to test the original rule ( $w_{max} \in \{24, 44, 84\}$ ).
- Maximum Ensemble Size ( $r_{max}$ ): Sweeping values from 10 to 175 to analyze the relationship between ensemble size and accuracy, particularly for long-series datasets.
Adaptive Rule Proposal: Based on the sensitivity analysis, the authors proposed a new rule for $r_{max}$ that depends on series length ( $n$ ) and the number of classes ( $c$ ), rather than the original rule based on training set size ( $m$ ) and series length.

Key Results

Reproduction: The independent reproduction achieved a mean accuracy of 0.865 and a median of 0.928, closely matching the original paper's values (0.863 and 0.927). A Wilcoxon signed-rank test confirmed no statistically significant difference ( $p = 0.655$ ). WEASEL 2.0 was confirmed as the most accurate dictionary classifier, outperforming WEASEL 1.0 and cBOSS, and performing comparably to top kernel-based methods like MultiRocket.
Sensitivity Findings:
- Robust Choices: The choice of Ridge classifier, the absence of feature weighting, and the maximum window-size heuristic were found to be empirically robust. Replacing Ridge with SGD significantly reduced accuracy ( $-6.9\%$ mean) and increased fit time. Applying TF-IDF or sublinear TF scaling either reduced accuracy or increased memory usage without accuracy gains. Varying $w_{max}$ around the original tiers yielded no significant accuracy improvements.
- Over-Provisioned Ensemble: The maximum ensemble-size heuristic was found to be over-provisioned for long-series datasets. For datasets with series length $n > 500$ , increasing $r_{max}$ beyond 75 resulted in marginal or negative accuracy returns while incurring substantial memory costs (hundreds of MBs).
Adaptive Rule Performance: The proposed adaptive rule sets $r_{max}$ $r_{ma x}$ based on series length and class count:
- $r_{max} = 75$ if $n > 700$
- $r_{max} = 25$ if $c \le 2$
- $r_{max} = 50$ otherwise
  Evaluated on 112 fixed-length datasets, this rule reduced peak fit memory by a median of 37 MB (mean 395 MB) and fit time by a median of 0.4 s (mean 4 s). Crucially, the median accuracy change was 0%, with a mean reduction of only -0.11%. The savings were concentrated on long-series datasets where the original rule allocated the largest ensembles.

Significance and Claims
The paper claims to provide the first independent reproduction of WEASEL 2.0, validating its status as a state-of-the-art dictionary classifier. By systematically testing under-justified design choices, the authors demonstrate that three of the four heuristics are robust, while the fourth (ensemble size) is inefficient for specific data regimes. The primary contribution is the introduction of an adaptive ensemble-size rule that significantly reduces memory and time requirements for long-series datasets with negligible impact on accuracy. The authors position this work as a complementary approach to proposing new models, instead focusing on refining existing classifiers to improve efficiency and resource management. They note that future work could extend this rule to multivariate settings or develop data-driven, cross-validated versions of the threshold selection.

More like this