An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection

Title: An Insight on Evaluation Metrics Under the Imbalanced Case of Anomaly Detection

Problem Statement
Anomaly detection is inherently characterized by severe class imbalance, where anomalies are rare compared to normal data. This imbalance complicates the interpretation of standard evaluation metrics. While metrics such as AUROC, AUPR, F1-score, and Matthews Correlation Coefficient (MCC) are widely used, their numerical values convey different meanings depending on the anomaly ratio. A single scalar score often discards information regarding model behavior, and identical scores can correspond to vastly different classifier behaviors under varying degrees of imbalance. Furthermore, in unsupervised settings where training occurs only on normal data, evaluation is conducted under highly skewed test distributions, making performance assessment delicate. Although weighted variants of standard measures exist, they are often avoided in anomaly detection to prevent decoupling evaluation from the operational class distribution. Consequently, there is a lack of consensus on how to interpret these standard metrics across different imbalance regimes.

Methodology
The authors propose a systematic analysis of four common anomaly detection metrics (AUROC, AUPR, F1, and MCC) across a wide range of class imbalance ratios. The core methodological contribution is the introduction of metric landscapes, a visualization framework that maps metric scores over the space of True Positive Rate (TPR/Recall) and True Negative Rate (TNR/Specificity).

The study utilizes:

1. Datasets: A diverse set of benchmarks representing Time-Series Anomaly Detection (TSAD), One-Class Classification (OCC), and Image Anomaly Detection (IAD). This includes UCR, WaDi, PUMP, SWaT, PSM, CIFAR-10, ViSA, and MVTec, covering anomaly ratios from 0.19% to 90%.
2. Null Analysis: Establishing baselines by computing metrics over 100 runs of random predictions across imbalance ratios (from 99:1 to 1:99) to determine mean, standard deviation, and mean positive deviation.
3. Landscape Evolution: Visualizing the difference between metric landscapes in balanced (50:50) and imbalanced (e.g., 5:95) settings to observe how metric preferences shift.
4. Factor Analysis: Investigating latent relationships between metrics using Tucker's congruence coefficients and Pearson correlations across three imbalance groups (Low, Medium, High).

Key Results

• Metric Stability and Baselines:
  • AUROC and MCC remain centered around 0.5 across all imbalance ratios under random predictions, though AUROC exhibits higher variability (mean positive deviation up to ~0.25).
  • F1 and MCC show low variance under random behavior, making them less prone to spurious high scores.
  • AUPR baseline varies linearly with the anomaly proportion, making deviations from random performance easy to quantify, though it shows non-negligible upward deviations in specific regimes.
• Landscape Behavior:
  • AUROC is invariant to class imbalance; its landscape structure remains unchanged regardless of the ratio.
  • AUPR and F1 are highly sensitive to imbalance. As the negative class dominates, AUPR landscapes decrease significantly (by 0.2–0.4 in many regions), reflecting reduced influence of negative-class performance. F1 requires increasingly high TNR to achieve the same score as the anomaly ratio decreases.
  • MCC exhibits a complex pattern where extreme TNR values remain stable, but the distribution compresses around mid-range values, making average score interpretation less direct under severe imbalance.
• Minimal Detection Requirements:
  • For AUROC, no positive samples need to be detected to achieve a score of 0.5.
  • For AUPR, achieving high scores in balanced datasets requires 100% correct classification of negatives even with minimal positive detection.
  • For F1, increasing the score always requires more correctly classified positives, but the requirement for correctly classified negatives grows as the anomaly ratio decreases.
  • For MCC, achieving a score of ~0.4 on highly imbalanced datasets (e.g., UCR) is possible without detecting any positives and with only ~10% of negatives correctly classified, though this remains below the random baseline.
• Metric Relationships:
  • In Medium imbalance regimes, all metrics exhibit similar loading profiles and high correlations, suggesting they capture related structures.
  • In Low (highly imbalanced) regimes, metrics become more complementary. AUROC and MCC remain closely related, while F1 and AUROC show the weakest agreement.
  • In High imbalance regimes (positive class dominant), metrics decorrelate further. Notably, AUROC and MCC maintain total similarity (Tucker congruence of 0.997), whereas AUPR and AUROC show reduced congruence.

Significance and Contributions
The paper argues that existing metrics remain useful provided their interpretation explicitly accounts for the dependency between metric values, prediction quality, and class imbalance. The primary contribution is the introduction of metric landscapes as an intuitive framework to visualize how different metrics emphasize different trade-offs between sensitivity and specificity and how these preferences shift under increasing imbalance.

The work provides practical guidance for interpreting and comparing anomaly detection results across datasets with different imbalance ratios. It does not propose a single "best" metric for all scenarios but rather offers a reference point for understanding metric behavior under realistic conditions. The authors note that identical scores can correspond to different classifier behaviors, and their analysis helps clarify these distinctions. Future work is identified as extending this analysis to other domains such as industrial monitoring, IoT, cybersecurity, and autonomous driving.