Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles

Technical Summary: Customer Churn Prediction on Structured Data Using FT-Transformer and Stacking Ensembles

Problem Definition
Customer churn prediction in structured tabular datasets presents distinct challenges compared to unstructured data (images, text). Key difficulties include severe class imbalance (typically 15–25% churners), complex nonlinear feature interactions, and heterogeneous feature types (numerical and categorical). While tree-based ensembles (e.g., XGBoost) have historically outperformed deep learning on tabular data due to their ability to handle axis-aligned splits and irregular decision boundaries, recent transformer architectures offer potential for capturing higher-order feature dependencies. However, prior research has largely treated these model families in isolation, often lacking rigorous statistical validation, probability calibration, and systematic ablation of hybrid components. Furthermore, many existing studies rely on synthetic oversampling (e.g., SMOTE), which can introduce artifacts and distort minority class distributions.

Methodology
The authors propose a hybrid architecture that integrates FT-Transformer (Feature-Tokenized Transformer) with XGBoost via a calibration-aware stacking ensemble. The framework operates as follows:

1. Data Preprocessing: The pipeline employs Z-score normalization and median/mode imputation derived strictly from the training set to prevent data leakage. Crucially, the authors avoid synthetic oversampling, opting instead for class-weighted loss functions to handle the ~4:1 class imbalance.
2. Base Models:
   • FT-Transformer: Converts heterogeneous features into a sequence of embeddings. Numerical features undergo linear embedding, while categorical features use learned embedding matrices. A learnable [CLS] token aggregates information, and multi-head self-attention mechanisms model pairwise feature interactions. The model uses a class-weighted binary cross-entropy loss.
   • XGBoost: A gradient-boosted decision tree ensemble configured with early stopping and a scale_pos_weight parameter to address imbalance.
3. Stacking Ensemble: The outputs of the two base models are combined using Out-of-Fold (OOF) predictions generated via 5-fold cross-validation (averaged over 5 random seeds). A Logistic Regression meta-learner is trained on these OOF predictions to learn optimal combination weights and recalibrate probabilities.
4. Validation: The study utilizes rigorous statistical validation, including 25 independent evaluations (5-fold CV × 5 seeds), paired t-tests, and Expected Calibration Error (ECE) analysis.

Key Contributions
The paper claims four primary contributions:

1. Hybrid Architecture: It is among the first to systematically combine FT-Transformer and XGBoost specifically for churn prediction, validating error independence ($\rho = 0.62$) and probability calibration (ECE = 0.038).
2. Comprehensive Ablation: The study isolates the contributions of the transformer layers, the ensemble strategy, and the meta-learner selection, demonstrating that both components are essential and non-redundant.
3. Probability Calibration: Through ECE analysis, the framework demonstrates improved decision reliability, showing that the meta-learner effectively recalibrates the overconfident outputs of the base models without requiring separate calibration datasets.
4. Reproducibility: The authors provide detailed algorithmic specifications, preprocessing steps, and exact hyperparameter configurations to facilitate independent validation.

Experimental Results
Evaluated on a public bank churn dataset (10,000 customers, 20.4% churn rate), the proposed hybrid model achieved:

• F1 Score: 62.10% (95% CI: [61.65, 62.55])
• AUC-ROC: 0.861 (95% CI: [0.858, 0.864])
• PR-AUC: 0.647

These results represent statistically significant improvements ($p < 0.001$) over strong baselines, including a Multi-Layer Perceptron (MLP) (+3.37 F1 points), XGBoost (+3.89 F1 points), and the FT-Transformer alone (+1.10 F1 points).

Feature Importance and Interpretation
Both SHAP values (from XGBoost) and attention weights (from FT-Transformer) identified Age, IsActiveMember, and NumOfProducts as the most critical predictors. The transformer successfully captured specific interactions, such as the sharp increase in churn risk for older customers holding three or more products (42% churn vs. 12% for younger customers with one product), an interaction pattern that linear models fail to represent.

Significance and Claims
The paper posits that its significance lies in bridging recent advances in attention-based modeling with established ensemble techniques. The authors argue that the hybrid approach leverages complementary inductive biases: transformers capture continuous feature interactions and global dependencies, while tree-based models excel at discrete decision boundaries.

The study emphasizes that the ensemble's success is not merely due to model complexity but stems from partial error independence ($\rho = 0.62$) between the base models, allowing the meta-learner to recover 89.1% of cases where one model failed. Furthermore, the framework offers a reproducible reference architecture that addresses gaps in statistical validation and probability calibration found in prior literature.

Limitations and Scope
The authors maintain a modest stance regarding generalization. They acknowledge that the study is based on a single dataset (European banking, 2010–2015) and lacks temporal behavioral signals. They caution that production performance may degrade by 1–2 percentage points on unseen data and that the framework requires domain-specific feature alignment for transfer to other sectors like telecommunications or insurance. The paper explicitly avoids claiming universal optimality, noting that for datasets with very small sample sizes (<2,000) or strict latency constraints (<10ms), gradient boosting alone may be preferable.