Technical Summary: Benchmarking Binary Classifiers Under Class Imbalance Without Rebalancing Techniques

Problem Statement

Class imbalance presents a critical challenge in supervised classification, particularly in high-stakes domains such as medical diagnostics, fraud detection, and anomaly detection where minority class instances are rare. While the machine learning community has extensively explored explicit rebalancing techniques (e.g., SMOTE, undersampling, cost-sensitive learning) to mitigate this issue, there is a notable gap in understanding the inherent robustness of binary classifiers when these techniques are not applied.

Existing literature often assumes that rebalancing is necessary for effective learning, yet few studies have systematically evaluated classifier performance under extreme imbalance conditions—specifically in one-shot (single minority instance) and few-shot scenarios—without external intervention. Furthermore, the interplay between data complexity (decision boundary geometry) and the severity of class imbalance remains under-explored. This study addresses these gaps by assessing the "as-is" performance of diverse binary classifiers to determine the minimum imbalance ratio that still permits reliable classification and generalization.

Methodology

The authors propose a systematic experimental framework designed to evaluate classifier resilience without explicit rebalancing. The methodology consists of three primary strategies:

1. Systematic Reduction of Minority Class

The study employs a controlled reduction of minority class samples across real-world datasets.

• Procedure: Starting from the original dataset, the proportion of minority class training examples is systematically reduced to 100%, 50%, 25%, 10%, 5%, and 1%.
• Extreme Scenarios: For reductions below 1%, the training set retains only a single representative minority datapoint (one-shot scenario).
• Control: The majority class samples remain intact in all folds to isolate the effect of minority scarcity.
• Validation: Experiments utilize a 2×5-fold stratified cross-validation strategy to ensure stable performance estimates.

2. One-Shot and Few-Shot Baselines

To simulate real-world data scarcity where acquiring minority samples is difficult or expensive, the authors construct specific training sets:

• Scenarios: Training sets are constructed with k = 1, 3, and 5 minority class instances.
• Repetition: To mitigate bias from specific sample selection, experiments are repeated ten times with random sampling of minority instances, reporting average AUC-ROC and AUC-PR scores.
• Comparison: These results are compared against One-Class Classification (OCC) methods to evaluate performance when no minority data is available for training.

3. Synthetic Decision Boundary Generation

To decouple imbalance effects from data complexity, the authors generate synthetic datasets representing varying geometric structures:

• Linear Separable: High class separation.
• Moderate Non-Linear: Clusters with noise.
• Non-Linear Redundancy: High-dimensional redundancy reduced via PCA.
• Gaussian Quantiles: Circular class structures with overlapping boundaries.
• Hard Non-Linear (XOR): Complex, non-linear decision boundaries.
  These datasets allow for the evaluation of how geometric complexity interacts with skewed class distributions.

Models and Metrics

• Classifiers Evaluated: A diverse set including traditional models (Decision Tree, k-NN), margin-based classifiers (SVM), ensemble methods (Random Forest, BaggingRF, RUSBoost, XGBoost, LightGBM, CatBoost), and advanced deep tabular models (TabPFN).
• Evaluation Metrics: Performance is assessed using AUC-ROC, AUC-PR, Accuracy, Precision, Recall, F1-score, and G-mean.
• Interpretability: Cumulative Distribution Function (CDF) plots of prediction scores are used to visualize the separation between majority and minority class predictions.

Key Results

1. Impact of Systematic Reduction

• Performance Degradation: As the minority class size decreases, most classifiers (DT, k-NN, RF, XGBoost) exhibit a consistent and severe drop in AUC-ROC and AUC-PR, particularly below the 25% threshold.
• Robust Models: TabPFN consistently demonstrates the highest robustness across all imbalance levels, including extreme one-shot scenarios. CatBoost and LightGBM also maintain relatively stable performance down to 10% minority representation.
• Traditional Failure: Traditional models like Decision Trees and k-NN deteriorate rapidly when minority representation falls below 25%, often performing near random chance in one-shot settings.

2. One-Shot and Few-Shot Performance

• Scarcity Sensitivity: Most classifiers fail to generalize effectively with only 1 minority sample. However, increasing the sample count from 1 to 3 or 5 significantly improves performance for many models.
• TabPFN Dominance: TabPFN maintains strong performance even in one-shot settings, outperforming ensemble methods and traditional classifiers.
• OCC Utility: One-Class Classification models, while inherently limited by their reliance on majority data, showed surprising robustness on certain datasets, serving as a viable fallback when minority data is entirely inaccessible.

3. Influence of Data Complexity

• Geometry Matters: Classifier performance is heavily dependent on the underlying data geometry.
  • Linear/High Separation: Reliable classification is achievable even at extreme imbalance ratios if the decision boundary is simple.
  • Complex/Non-Linear: As decision boundaries become more complex (e.g., XOR patterns, overlapping Gaussian quantiles), performance degrades sharply for all models except the most advanced (TabPFN, CatBoost).
• Synthetic Findings: In synthetic datasets with high redundancy or non-linearity, only advanced models retained acceptable performance under few-shot conditions.

4. Prediction Score Analysis

CDF plots reveal that TabPFN generates well-separated distributions for majority and minority classes, maintaining high confidence even under severe imbalance. In contrast, traditional models like DT show significant overlap in prediction scores, indicating weak discriminative ability.

Significance and Contributions

The paper claims the following contributions to the field of imbalanced learning:

1. Intrinsic Robustness Assessment: The study provides a rigorous evaluation of binary classifiers "as-is," without the confounding variables of rebalancing techniques. This clarifies the inherent capacity of specific algorithms to handle scarcity.
2. Identification of Thresholds: The research identifies the minimum imbalance ratios at which reliable classification is still achievable, noting that this threshold varies significantly by model and data complexity.
3. Model Selection Guidance: The findings offer practical guidance for practitioners in low-resource or high-stakes domains (e.g., medical diagnostics), suggesting that advanced models like TabPFN and specific boosting ensembles (CatBoost) are superior choices when rebalancing is not feasible or desirable.
4. Interpretability: By utilizing decision boundary visualization and CDF analysis, the study offers deeper insights into how classifiers separate classes under extreme imbalance, moving beyond simple metric reporting.

Limitations

The authors acknowledge that the study is based on benchmark datasets and controlled synthetic scenarios. Hyperparameters were restricted to standard configurations, and the results may vary in larger-scale or domain-specific applications. The authors suggest that future work should explore adaptive hyperparameter tuning and privacy-preserving evaluation in sensitive domains.

In conclusion, the paper argues that while class imbalance severely challenges traditional classifiers, advanced models possess a significant intrinsic ability to generalize under scarcity, offering a viable path for classification in scenarios where explicit rebalancing is impractical.