Classifier Pooling for Modern Ordinal Classification

1. Problem Statement

Ordinal classification (or ordinal regression) involves predicting labels that have a natural, ordered hierarchy (e.g., disease stages, Likert scales, cancer grades). While traditional statistical methods like Ordered Logit and Ordered Probit models exist, they are often limited by:

• Simplicity: They rely on linear assumptions and convex optimization, lacking the capacity to model complex, non-linear relationships found in modern datasets.
• Lack of Modern Integration: Powerful machine learning classifiers (e.g., Support Vector Machines, Gradient Boosting, Naïve Bayes) are rarely adapted for ordinal tasks.
• Inappropriate Paradigms: When modern classifiers are applied to ordinal data, they often default to:
  • Multiclass Classification: Treating classes as nominal (ignoring order), which discards valuable ranking information.
  • Regression: Treating classes as continuous numbers, which incorrectly assumes linear distances between ordinal steps (e.g., assuming the gap between Stage 1 and 2 is identical to Stage 2 and 3).

Furthermore, existing model-agnostic pooling paradigms (Cumulative and Hierarchical) were not previously available in accessible software libraries, leading to their underutilization.

2. Methodology

The authors propose a model-agnostic framework that allows any standard binary classifier to be adapted for ordinal classification. They introduce two primary paradigms implemented in a new Python package called statlab:

A. Core Algorithms

The framework relies on Algorithm 1 for training, where a set of binary classifiers is trained to predict whether a sample exceeds a specific threshold between adjacent classes ($P(Y > c_i)$).

1. Difference-Based Ordinal Classification (Cumulative):
   • Training: Trains $N-1$ binary classifiers, each predicting $P(Y > \text{threshold}_i)$.
   • Inference (Algorithm 2): Estimates the probability of a specific class by calculating the difference between the cumulative probabilities of adjacent thresholds.
   • Constraint: Applies a monotonic constraint to ensure probabilities are non-increasing as the threshold increases.
2. Tree-Based Ordinal Classification (Hierarchical):
   • Training: Similar to the difference method, but focuses on a "best split index" to structure the inference.
   • Inference (Algorithm 3): Uses a conditional probability approach (Equation 1). It iteratively conditions the probability of a class on the predictions of adjacent classifiers, moving from a "best split" index outward. This mimics a decision tree structure where probabilities are conditioned on previous splits.

B. Hyperparameter Optimization

A critical component is selecting the Best Split Index (the threshold used as the pivot for inference). The paper evaluates four strategies:

1. Arbitrary: First or last threshold.
2. Middle Index: The center threshold.
3. Even Split: The threshold where the dataset is most balanced (equal number of samples above and below).
4. Best Classifier: Selecting the threshold that yields the highest F1-score on a validation set (implemented via 4-fold cross-validation).

C. Software Implementation

The authors developed the statlab Python package, which wraps any sklearn-compatible classifier (requiring fit, predict, and predict_proba methods). It also includes a BaseOrdinalClassifier for models that do not natively output probabilities.

3. Key Contributions

1. Model-Agnostic Framework: A generalizable method to convert any modern binary classifier into an ordinal classifier without modifying the underlying algorithm.
2. Open-Source Software: The release of the statlab package on PyPI, providing the first accessible implementation of cumulative and hierarchical ordinal classification paradigms for modern ML.
3. Comprehensive Evaluation: A rigorous benchmark across six diverse datasets (clinical, automotive, wine quality, radiomics, and abalone rings) comparing:
   • Native multiclass classifiers.
   • Traditional Ordered Logit/Probit models.
   • The proposed Difference and Tree-based ordinal methods.
4. Hyperparameter Analysis: Systematic testing of "Best Split Index" selection methods to guide practitioners.

4. Results

The experiments were conducted across three main scenarios:

• Experiment 1 (Cross-Domain Performance):

  • The proposed ordinal methods (Difference and Tree-based) outperformed or matched native multiclass classifiers in 75–79% of metrics across ordinal datasets.
  • They frequently surpassed traditional Ordered Logit/Probit models, particularly on complex datasets like RetinaMNIST radiomics.
  • Negative Control: On the MNIST dataset (digits 0–9, treated as non-ordinal), the ordinal methods performed worse or equal to chance, confirming that forcing ordinality on non-ordinal data degrades performance.

• Experiment 2 (Data Scarcity and Class Complexity):

  • As dataset size decreased and the number of classes increased, performance generally dropped for all methods.
  • However, the proposed ordinal methods showed greater robustness, with smaller performance degradation compared to native classifiers. This suggests they are particularly effective for small datasets or problems with many outcome classes.

• Experiment 3 (Split Index Selection):

  • The "Middle Index" strategy performed best overall (achieving the highest metric in 7/24 cases).
  • The "Best Classifier" (validation-based) method did not consistently outperform simpler heuristics, suggesting the computational cost of cross-validation for index selection may not always be justified.

5. Significance and Limitations

Significance:

• Bridging the Gap: The work successfully bridges the gap between classical ordinal statistics and modern machine learning, allowing practitioners to leverage powerful algorithms (like Gradient Boosting) for ordered data.
• Practical Utility: The methods are particularly valuable in clinical settings (e.g., pathology staging) where data is often sparse, and the ordinal nature of the outcome is critical.
• Reproducibility: By providing open-source code, the authors facilitate the adoption of these methods in the broader scientific community.

Limitations:

• Deep Learning: The approach is less suited for deep learning (neural networks), which may inherently learn ordinal structures through complex decision boundaries.
• Dataset Dependence: As with all ML, performance varies significantly by dataset. No single method dominates all scenarios.
• Probability Calibration: While the methods improve accuracy, the calibration of the resulting probabilities requires careful interpretation, especially in the Tree-based approach.

In conclusion, the paper demonstrates that treating ordinal data with specialized, model-agnostic pooling strategies yields superior performance compared to standard multiclass or regression approaches, especially in data-scarce environments, and provides the necessary tools to implement these solutions easily.