Technical Summary: macrOData – New Benchmarks of Thousands of Datasets for Tabular Outlier Detection

Problem Statement

Outlier detection (OD) on tabular data is critical for applications in fraud detection, healthcare, and environmental monitoring. However, the field suffers from a lack of large-scale, diverse, and standardized benchmarks, making it difficult to assess scientific progress or make informed methodological choices. The current de facto standard, ADBench, is limited to only 57 datasets. The authors argue that this small scale restricts statistical power, reduces diversity, and makes empirical comparisons highly sensitive to dataset selection.

Furthermore, the paper identifies a specific structural limitation in ADBench: it predominantly captures global outliers that resemble Gaussian noise. This characteristic inadvertently favors simple, global distance-based methods (like KNN) and recent foundation models pretrained on synthetic Gaussian mixtures (like FoMo-0D), potentially masking their inability to handle more complex, semantic, or local anomalies. The unsupervised nature of OD, combined with high hyperparameter sensitivity, exacerbates these evaluation issues, as tuning choices can drastically impact performance without robust statistical validation.

Methodology: The macrOData Suite

To address these gaps, the authors introduce macrOData, a comprehensive benchmark suite comprising 2,446 datasets curated from three distinct components. The data was sourced primarily from Tablib (627 million tables) and various classification repositories.

1. OddBench (Semantic Anomalies)

• Source: A curated subset of Tablib containing real-world tables where anomalies are defined by semantic meaning (e.g., "fraud," "failure," "defect").
• Curation: The authors applied a rigorous five-step filtering process:
  1. Preliminary Filter: Removed tables with high null/infinity ratios, monotonic features, or insufficient size (<1,000 samples, <3 numerical features).
  2. Target-based Filter: Selected tables with a categorical feature explicitly identifying anomalies (e.g., "attack," "error").
  3. Hash and Separability Check: Removed duplicates and ensured anomalies were separable from inliers (Random Forest classifier ROC-AUC > 0.6).
  4. Manual Quality Control: Manual inspection to verify semantic relevance and remove remaining duplicates.
  5. Quality Improvements: Added standardized train/test splits, enriched metadata, and created a representative subset.
• Scale: 790 datasets (690 public, 100 private).

2. OvRBench (Statistical Outliers)

• Source: Derived from the remaining half of the filtered Tablib data and established classification benchmarks (OpenML-CC18, TabZilla, TabRepo, etc.).
• Methodology: Uses a One-vs-Rest (OvR) approach where a single class is designated as the inlier and all other classes are treated as outliers. This captures concept shifts and out-of-distribution (OOD) detection scenarios.
• Target Selection: For Tablib data without predefined targets, the authors filtered for features named "label" or "target" with cardinality between 2 and 10.
• Scale: 856 datasets (756 public, 100 private).

3. SynBench (Synthetic Priors)

• Source: 800 synthetically generated datasets designed to span diverse data priors and outlier archetypes.
• Generators:
  • Gaussian Mixture Models (GMM): Captures multi-modal structures; outliers generated by inflating variances in subspaces.
  • Structural Causal Models (SCM): Generates non-linear, non-Gaussian distributions; outliers created via measurement noise inflation or structural graph modifications.
  • Copulas: Models diverse marginals and dependencies; outliers created by pushing coordinates to boundaries or disrupting joint structures.
• Scale: 800 datasets (all public).

Benchmark Features

• Standardization: All datasets include standardized train/test splits (50% inliers for training; 50% inliers + all outliers for testing).
• Public/Private Partition: Private subsets (100 datasets per real-world benchmark) have redacted test labels to support an online leaderboard for blinded, fair evaluation.
• Metadata: Datasets are annotated with semantic tags, domain keywords, and source links, generated using Large Language Models (LLMs).

Experimental Evaluation

The authors conducted extensive experiments evaluating 14 OD methods across all 2,446 datasets, covering classical, deep, and foundation models.

• Methods Evaluated:
  • Classical: OCSVM, KNN, LOF, CBLOF, IForest, EGMM, DTE-NP.
  • Deep: GOAD, ICL, DTE-C, NPT-AD.
  • Foundation Models: FoMo-0D, OutFormer, TabPFN-OD.
• Hyperparameter (HP) Strategy: To ensure fair evaluation, the authors evaluated methods across extensive HP configurations. Classical models were tested on full grids; deep models were sampled from predefined spaces; foundation models required no tuning.
• Metrics: Performance was measured using AUROC and AUPRC, aggregated via Average Rank, ELO scores, Win Rate, rescaled AUC, and Champion Delta. Statistical significance was determined via permutation tests.

Key Results

1. Foundation Models Dominate: Foundation models (specifically OutFormer and FoMo-0D) significantly outperformed all classical and deep learning methods in terms of detection performance (AUPRC) and runtime. They occupy the Pareto front of the performance-time trade-off.
2. Classical vs. Deep: Classical methods generally outperformed deep learning models, largely due to the latter's high sensitivity to hyperparameter choices. Among classical methods, EGMM, DTE-NP, and LOF showed the best performance-runs trade-offs.
3. Hyperparameter Sensitivity: Deep models exhibited high variance in performance across different hyperparameter configurations. In contrast, ensemble methods (like EGMM) and foundation models demonstrated greater robustness, with foundation models eliminating the need for tuning entirely.
4. ADBench Limitations Confirmed: The results validated the authors' hypothesis that ADBench's bias toward Gaussian-like global outliers allows simple methods (KNN) and Gaussian-pretrained foundation models to achieve state-of-the-art results, which may not generalize to more complex, real-world semantic anomalies.

Significance and Contributions

The paper claims the following contributions:

• "Demystifying" ADBench: The work critically analyzes the limitations of the current standard benchmark, revealing its bias toward Gaussian noise and global outliers, which skews the perceived effectiveness of modern algorithms.
• New Large-Scale Benchmarks: The introduction of macrOData provides the first benchmark suite of its scale (2,446 datasets) with diverse semantic and statistical outliers, standardized splits, and public/private partitions.
• Comprehensive Evaluation & Guidelines: The extensive evaluation establishes that foundation models are currently the most effective and efficient approach for tabular OD, offering plug-and-play capabilities without hyperparameter tuning.
• Research Enabler: By providing a robust, diverse testbed, macrOData aims to foster future research in meta-learning and foundation models for OD, moving the field beyond the constraints of small-scale, low-diversity datasets.

The authors emphasize that macrOData is open-sourced, including code for curation, dataset construction, and individual performance results, alongside a publicly accessible leaderboard to track future progress.