Meta-classification of one-class classification models using ranking correlation and nearest neighbor

Technical Summary: Meta-classification of One-Class Classification Models

Problem Statement
While Machine Learning (ML) techniques are widely applied to various domains, the application of ML to ML models themselves remains an unexplored direction. Specifically, this paper addresses the challenge of classifying One-Class Classification (OCC) models. OCC is a problem where the training set consists solely of one class. The authors posit that all ML models can be approximated as OCC models (e.g., binary or multi-class models can be viewed as sets of OCC models). The core difficulty lies in the fact that OCC models possess unique score functions and thresholds that vary by algorithm, making direct comparison of their internal mechanics impossible. The goal is to develop a method to classify these models based on their behavior rather than their internal parameters.

Methodology
The proposed framework treats OCC models as data points to be classified. The methodology proceeds in three main stages:

1. Representation as Normality Rankings:
   Instead of comparing raw score functions (which differ in variables and scales across algorithms), the method constructs "normality rankings." A separate "ranking set" (a set of independent samples) is fed into the OCC models. Each model outputs a score for every sample in this set. These scores are then converted into ranks (orderings) for each model. This transforms the model into a vector of ranks, creating a common representation space regardless of the underlying algorithm.

2. Similarity Measurement:
   To compare two OCC models, the paper utilizes ranking correlation metrics. Specifically, it employs Spearman correlation (for speed) and Kendall Tau correlation (for handling ties) to measure the similarity between the normality rankings generated by different models.

3. Classification via Nearest Neighbor:
   The classification task is performed using a Nearest Neighbor (NN) approach. Since NN is a "lazy learner," the training phase involves storing the rankings of labeled OCC models. During the testing phase, the ranking of an unknown model is compared against the stored training rankings using the correlation metrics. The unknown model is assigned the label of the most similar (nearest) training model.

Key Contributions

• First Journal Article on ML Model Classification: The authors claim this is the first journal article to explicitly classify ML models, specifically focusing on OCC models.
• Unified Solution: The method provides a unified framework for classifying three distinct entities: OCC models, the datasets used to train them, and rankings themselves.
• Theoretical Insight: The paper establishes that classifying OCC models is essentially equivalent to classifying the datasets that trained them, as there is a one-to-one correspondence between an OCC model and its training dataset.
• Extended Experiments: Unlike a previous conference version, this work includes detailed experiments on classifying ensemble models, data balance (normal vs. abnormal ratios), and hyperparameters.

Experimental Results
The experiments utilized four base OCC algorithms (OCSVM, LOF, Isolation Forest, GMM) and their ensembles, trained on subsets of the KDD Cup dataset.

• Dataset Classification: The method achieved 100% accuracy in distinguishing models trained on "normal" data versus "abnormal" data. It also achieved high accuracy (99.3–99.4%) in distinguishing models trained on mixed data. Classifying specific normal-to-abnormal ratios (11 classes) yielded accuracies of 83.6% (leave-one-out) and 87.3% (2-fold).
• Algorithm Classification: When distinguishing between the four base algorithms, the method achieved 100% accuracy. However, accuracy dropped when classifying ensemble models (15 classes), achieving 85.5% on normal data and significantly lower (23.7%) on abnormal data, suggesting ensemble models are more similar to each other than base learners.
• Hyperparameter Classification:
  • Categorical Hyperparameters: High accuracy was achieved for categorical parameters (e.g., OCSVM kernels: 93.8–100%; GMM covariance types: 75%).
  • Numerical Hyperparameters: Accuracy was low for numerical parameters (e.g., LOF neighbors: 2.8%; IF max features: 27.9%). The authors note that while classification accuracy is low, the results resemble a regression problem where predictions are close to actual values, suggesting minor numerical changes do not drastically alter the model's ranking behavior.
• Real-World Applications:
  • Biometric Signals: The method was applied to Ballistocardiogram (BCG) signals, successfully detecting anomalies (breath-holding, position changes) by analyzing shifts in ranking correlations.
  • Sleep Records: In an unsupervised setting using sleep records from 1308 participants, the method identified "abnormal" sleep behaviors by treating individual user datasets as OCC models and detecting outliers in the ranking space.

Significance and Limitations
The paper claims that the classification of OCC models offers a new perspective in Explainable AI (XAI), specifically as a method to analyze black-box models by their output rankings. It suggests that classifying datasets (treating multiple samples as a single input) may be easier than traditional sample-by-sample classification due to the greater differences between datasets.

However, the authors are modest about the limitations:

• Scalability: The approach faces scalability issues with large datasets due to the computational cost of computing ranking correlations (O(n) for Spearman, O(n²) for Kendall Tau).
• Black-Box Constraints: The method requires access to score functions and a known input shape to generate the ranking set. In real-world scenarios where only classification outputs are available or input shapes are unknown, creating a ranking set is challenging.
• Future Work: The paper explicitly states that classifying binary/multi-class models and developing "third-layer" models (classifying the classifiers of models) are left for future research.

In conclusion, the paper demonstrates that OCC models can be effectively meta-classified using ranking correlations and nearest neighbors, providing a unified approach to analyzing models, datasets, and rankings, with potential applications in anomaly detection and dataset characterization.