Technical Summary: Why Ranking Anomaly Detection Algorithms Isn't as Reliable as You May Think

Problem Statement

Anomaly detection (AD) is a safety-critical machine learning domain with applications in fraud detection, failure prevention, and network intrusion. Despite a proliferation of proposed algorithms—ranging from classical statistical methods to deep learning and foundation models—many new approaches claim state-of-the-art (SOTA) performance. However, these claims often rely on benchmark settings that are not aligned across studies. This lack of comparability raises significant concerns regarding the reproducibility and reliability of AD benchmarks. The core problem addressed is the instability of algorithm rankings: whether the "best" algorithm is a robust finding or an artifact of specific experimental design choices.

Methodology

To investigate the stability of AD rankings, the authors conducted a systematic analysis using the following framework:

• Algorithms: Seven representative, computationally lightweight AD algorithms were selected to allow for extensive repeated evaluations: KNN, LOF, Isolation Forest (IFOR), HBOS, PCA, CBLOF, and SEAN.
• Datasets: The study utilized 690 datasets from the OddBench benchmark suite. To manage computational costs (particularly for distance-based methods), datasets were limited to 5,000 randomly sampled instances.
• Rank Instability Metric: A novel metric, $\sigma_{rank}$, was introduced to quantify ranking stability. It is defined as the mean standard deviation of an algorithm's rank across varying benchmark settings:
  $$\sigma_{rank} = \text{mean}_{algo} \left( \text{std}_{setting} \left( \text{Rank}(algo, setting) \right) \right)$$
  A low value indicates stable rankings, while a high value suggests rankings are highly dependent on experimental design.
• Sources of Variability: The study isolated and simulated the impact of four specific factors by varying them while holding others constant:
  1. Dataset Selection: Varying the number and composition of datasets (subsets of 25, 100, and 400).
  2. Evaluation Metric: Comparing ROC-AUC vs. AUCPR.
  3. Hyperparameter Selection: Randomly sampling hyperparameter configurations.
  4. Random Seed Selection: Varying the random seed for data sampling and initialization.

Key Results

The analysis reveals that algorithm rankings in anomaly detection are highly unstable and sensitive to experimental choices:

• Rank Volatility: Rankings change substantially across different configurations. Remarkably, five of the seven studied algorithms achieved the top rank in more than 10% of all evaluated settings. Conversely, HBOS and CBLOF rarely appeared as the best-performing methods (HBOS once in 1,016 settings; CBLOF never).
• Feasibility of "SOTA" Claims: The results suggest it is comparatively easy to construct benchmark settings where a competitive algorithm appears state-of-the-art. In experiments, sampling as few as eight random settings was sufficient to find a configuration where a specific algorithm outperformed all others.
• Dominant Factors of Instability:
  • Dataset Selection and Hyperparameters: These are the primary drivers of ranking uncertainty. The choice of hyperparameters introduces substantial uncertainty, a factor the authors note is likely underestimated for classical methods and would be even more pronounced for modern deep learning methods with larger hyperparameter spaces.
  • Dataset Quantity: Increasing the number of datasets reduces instability, but the effect saturates. The study suggests that approximately 200 datasets are required to reduce total rank instability to a reliable level ($\sigma_{rank} \approx 0.2$).
  • Minor Factors: Random seed selection has a negligible impact, and the choice between common evaluation metrics (ROC-AUC vs. AUCPR) has a smaller, though noticeable, effect compared to dataset and hyperparameter choices.
• Magnitude of Uncertainty: The observed instability ($\sigma_{rank} \approx 0.34$) is significant given that the average ranking difference between the best and worst algorithms in the study was only 1.51 positions. An uncertainty of 0.34 ranks represents a large fraction of the total performance gap.

Significance and Recommendations

The paper argues that the notion of a universally "best" anomaly detection algorithm is often not robust and depends heavily on benchmark construction choices. Consequently, minor improvements in benchmark rankings should be interpreted with caution.

Based on these findings, the authors propose the following recommendations for future research:

1. Scale of Evaluation: Future benchmark studies should utilize at least 200 datasets to ensure ranking stability.
2. Hyperparameter Sensitivity: Benchmarks should explicitly evaluate algorithms under varying hyperparameter configurations rather than relying on a single manually selected setting.
3. Reporting Instability: Studies should include an instability score (such as the proposed $\sigma_{rank}$) to quantify the reliability of their conclusions.
4. Shift in Focus: The research community should prioritize robustness, reproducibility, and understanding the specific conditions under which algorithms perform well, rather than focusing primarily on marginal improvements in benchmark rankings.

The authors conclude that a tangible improvement in anomaly detection can only be assumed when major improvements are observed across a large amount of datasets and remain invariant to design choices.