Validation-Stage Combinatorial Fusion Analysis for Imbalanced Credit-Card Fraud Detection

Technical Summary: Validation-Stage Combinatorial Fusion Analysis for Imbalanced Credit-Card Fraud Detection

Problem Statement
Credit-card fraud detection presents a challenging machine learning problem characterized by extreme class imbalance, high costs of errors (both false positives and false negatives), and non-stationary fraud patterns. While strong gradient-boosted tree models (e.g., XGBoost, LightGBM) already serve as robust baselines for structured transaction data, the marginal value of additional ensemble or fusion methods is often unclear. This paper investigates whether Combinatorial Fusion Analysis (CFA)—a method that searches over model subsets and rank-score fusion rules—can yield reliable performance gains over the best single models and standard ensemble techniques (soft voting, stacking) on the IEEE-CIS Fraud Detection benchmark.

Methodology
The study employs a strict, leakage-free experimental protocol using a 60/20/20 train/validation/test split.

• Data & Preprocessing: The dataset comprises 590,540 anonymized transactions with mixed feature types. Preprocessing (handling missing values, label encoding, scaling) is fit exclusively on the training set.
• Base Classifiers: Seven distinct models serve as the candidate library: KNN, LDA, Logistic Regression, Random Forest, XGBoost, LightGBM, and TabNet. Linear models and KNN utilize PCA-reduced features, while tree ensembles and TabNet use the full feature set.
• Combinatorial Fusion Analysis (CFA):
  • Rank-Score Characteristic (RSC): Each model is characterized by its RSC function, mapping normalized rank positions to scores.
  • Diversity Measurement: Diversity between models is quantified as the average L1 distance between their RSC functions.
  • Search Space: The framework evaluates 120 model subsets (from the 7 base learners) combined with four fusion rules: Average Score, Weighted Score (diversity-weighted), Average Rank, and Weighted Rank. This results in 480 fusion configurations.
  • Selection Strategy: The optimal subset and fusion rule are selected based on performance metrics (AUC-ROC, AUPRC, F1) on the validation set. The test set is used only for final reporting.
• Auxiliary Experiments: The study includes a CTGAN augmentation experiment to test synthetic fraud generation and an ablation study on class imbalance ratios.

Key Contributions

1. Empirical Evaluation of CFA: The paper provides a rigorous evaluation of CFA on the IEEE-CIS benchmark, demonstrating its application with both traditional tree models and deep tabular learners (TabNet).
2. Subset Selection vs. Full Library: It establishes that the optimal ensemble is a compact subset of strong tree-based models (Random Forest, XGBoost, LightGBM) rather than the full model library or the most diverse set of models.
3. Statistical Rigor: The study utilizes bootstrap confidence intervals (1,000 resamples) to distinguish genuine performance gains from search noise, confirming that improvements over the best single model are statistically significant.
4. Negative Result on Augmentation: It reports a bounded negative result regarding CTGAN augmentation, showing that synthetic fraud generation degrades performance in this high-dimensional, mixed-type setting.

Results

• Performance: The best configuration, DEF WtScore (a diversity-weighted score fusion of Random Forest, XGBoost, and LightGBM), achieved:
  • AUC-ROC: 0.9405 (vs. 0.9343 for the best single model, LightGBM).
  • AUPRC: 0.6699 (vs. 0.6480 for LightGBM).
  • F1: 0.6373 (vs. 0.6126 for the best single model, XGBoost).
• Statistical Significance: Bootstrap confidence intervals for the performance differences exclude zero for all three metrics, confirming the gains are reliable.
• Comparison to Baselines:
  • CFA matches soft voting on AUC-ROC but improves AUPRC and F1.
  • Stacking performed worse than the best single model, likely due to the high correlation and strength of the base learners.
  • Score-based fusion outperformed rank-based fusion; the authors note that replacing informative scores with ranks removed useful separation in this context.
• Operational Metrics: At fixed operating points (e.g., 50% recall), the CFA ensemble maintained higher precision than LightGBM, indicating practical utility for fraud teams managing alert budgets.
• CTGAN & Imbalance: CTGAN augmentation reduced performance across all models and the CFA ensemble. Regarding class imbalance, CFA outperformed single models at fraud ratios of 1% and higher but underperformed at 0.5%, suggesting insufficient data to reliably estimate diversity signals at extreme sparsity.

Significance and Claims
The paper concludes that CFA is most valuable not as a mechanism to combine every available classifier, but as a validation-stage method for selecting a small, complementary subset of strong learners and assigning diversity-aware weights. The findings suggest that diversity is beneficial only when the added model contributes "useful errors" (complementary strengths) rather than merely different errors. The study cautions against the automatic use of synthetic data augmentation (specifically CTGAN) for high-cardinality fraud data and highlights that while well-tuned single models (like LightGBM) may suffice for broad ranking, ensemble methods offer meaningful, statistically significant gains for precision-recall trade-offs and operational thresholds.