🤖 machine learning

ChurnNet: A Optimized Modern AI for Churn Prediction

This study demonstrates that despite the advanced temporal modeling capabilities of the Unified Multi-Task Time Series Model, traditional machine learning algorithms like Random Forests, XGBoost, and Support Vector Machines consistently outperform it in customer churn prediction regarding accuracy, data efficiency, and computational cost.

Original authors: Syed Saad Saif, Giulio Maggiore, Paolo Russo, Damiano Distante

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Syed Saad Saif, Giulio Maggiore, Paolo Russo, Damiano Distante

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you own a busy grocery store. You have thousands of customers, but lately, some are quietly slipping away. They aren't sending you a formal "I quit" letter; they just stop coming in, or they visit less often. This is called churn. Your goal is to spot these customers before they disappear completely so you can offer them a coupon or a friendly hello to bring them back.

To do this, you need a smart computer program (an AI) to look at your sales records and predict who is about to leave.

This paper, titled "ChurnNet," is like a race between two different types of detectives trying to solve this mystery:

The "Old School" Detectives: These are established, reliable tools like Random Forests, XGBoost, and SVMs. Think of them as seasoned veterans who have been solving cases for decades. They are fast, don't need a huge library of evidence to work, and are great at spotting patterns in spreadsheets.
The "High-Tech" Detective: This is a new, fancy model called UniTS. Think of it as a brilliant, futuristic robot trained on massive amounts of data from all over the world. It's designed to understand complex time-based stories (like a customer's shopping habits over years) and is supposed to be the "state-of-the-art" solution.

The Big Question

The researchers asked: In the messy, real world of retail, does the fancy, high-tech robot actually do a better job than the reliable, old-school veterans?

How They Tested It

They didn't just guess; they set up a massive laboratory experiment:

The Evidence: They used four different sets of real-world shopping data (two from a private company, two from public archives).
The Rules: They created different scenarios. Sometimes the data was perfectly balanced (equal numbers of people leaving and staying). Sometimes it was messy and unbalanced (mostly people staying, with only a few leaving). They also tested the models on huge datasets and very small datasets.
The Definition of "Leaving": Since people don't send "I quit" letters in retail, the researchers had to invent a rule. They decided a customer is "churning" if their spending and visit frequency drop significantly compared to their own past habits.

The Results: The Surprise Winner

The results were a bit surprising to the tech world.

The Old School Detectives (XGBoost and Random Forest) won almost every time.

Accuracy: They were better at correctly identifying who was leaving.
Efficiency: They were much faster to train and didn't need a supercomputer to run.
Data Hunger: The high-tech robot (UniTS) was like a gourmet chef who needs a massive pantry to cook a good meal. When the researchers gave it a small dataset (like a tiny pantry), the robot's performance crashed. The old-school detectives, however, could cook a delicious meal even with very few ingredients.
The "Imbalance" Problem: In real life, most customers stay, and only a few leave. This is a "needle in a haystack" problem. The old-school detectives were much better at finding the needle (the churners) without getting confused by the haystack. The fancy robot sometimes got lost in the noise.

Why Did the High-Tech Robot Lose?

The paper suggests that the UniTS model is built to be a "universal" tool, meaning it's designed to handle any kind of time-series data (like stock markets, weather, or heartbeats). Because it's so complex and tries to learn everything at once, it needs massive amounts of data to work properly.

Retail data, however, is often "tabular" (rows and columns of numbers) and not always huge. In these specific conditions, the simpler, specialized tools (the old-school detectives) were actually more effective. They didn't need to be "smart" in a general sense; they just needed to be smart about this specific problem.

The Bottom Line

If you are a retailer trying to stop customers from leaving:

You don't necessarily need the most expensive, complex, "future-tech" AI.
The simpler, proven tools (like Random Forest and XGBoost) are often more accurate, faster, and cheaper to use, especially if you don't have millions of data points to feed a giant model.

The paper concludes that while fancy AI models are impressive, for the specific job of predicting retail customer churn, the "boring" old methods are still the champions.

Technical Summary: ChurnNet: An Optimized Modern AI for Churn Prediction

Problem Statement

The retail sector faces intensified competition and lowered switching barriers due to the shift toward omni-channel sales, making customer retention a strategic priority. Unlike contractual industries (e.g., telecommunications) where churn is defined by explicit cancellations, non-contractual retail churn is inferred from behavioral changes, such as reduced purchase frequency or volume. This "silent" defection is difficult to detect but critical to address.

While recent deep learning architectures, particularly transformer-based models like UniTS (Unified Multi-Task Time Series Model), have demonstrated impressive performance in binary time-series classification across various domains, their application to retail churn prediction remains under-investigated. These modern models are computationally expensive, prone to overfitting, and require large-scale datasets. Conversely, traditional machine learning (ML) methods are computationally efficient but may lack the capacity to model complex temporal dynamics. The central research question is whether the complexity of modern deep learning architectures is necessary for retail churn prediction, or if simpler, traditional models suffice to deliver strong predictive performance, data efficiency, and deployment viability.

Methodology

The study employs a structured pipeline to evaluate and compare traditional ML models against the UniTS transformer architecture across multiple real-world retail datasets.

Datasets and Pre-processing

The evaluation utilizes four datasets: two proprietary datasets from Infocube (IC1, IC2) and two public datasets (The Complete Journey [CJ], Let's Get Sort of Real [LGSR]).

Pre-processing: Transactional data is aggregated by customer ID, filtered to exclude non-positive values, and converted into weekly time series. Zero-padding is applied for weeks without transactions.
Filtering: Customers with fewer than 20 transactional weeks are excluded to ensure sufficient behavioral history.
Churn Labeling: A dynamic labeling strategy is used, defining an observation window ( $A=40$ weeks) and a labeling window ( $B=20$ weeks). A customer is labeled as a churner (1) if both their average weekly spending ( $\mu$ ) and transaction frequency ( $f$ ) drop below configurable thresholds ( $\alpha \cdot \mu_A > \mu_B$ and $\beta \cdot f_A > f_B$ ) in the labeling window. Three definitions (A, B, C) with varying sensitivity ( $\alpha, \beta \in \{0.3, 0.4, 0.5\}$ ) are tested.
Feature Windowing: A fixed 20-week feature window is used. For churners, this window precedes the inactivity period; for non-churners, a random 20-week segment is selected.

Experimental Design

The study constructs 36 distinct datasets by varying churn definitions and balancing strategies:

Balanced (1:1): Equal numbers of churners and non-churners.
Limited Unbalanced (1:4): All churners retained with four times the number of non-churners.
Balanced Subsampled (B-SS): Downsampled versions (2,500 or 5,000 customers) to test data efficiency.

Models Evaluated

Traditional ML: Random Forests (RF), XGBoost (XGB), and Support Vector Machines (SVM).
Modern AI: UniTS, a transformer-based model designed for multi-task time-series analysis. The study utilizes the supervised learning paradigm with transfer learning from a pre-trained checkpoint, avoiding extensive hyperparameter tuning to reflect "plug-and-play" usability.

Key Results

The empirical results, evaluated across accuracy, precision, recall, F1-score, and ROC-AUC, reveal consistent trends:

Superiority of Traditional Models: Tree-based models (XGBoost and Random Forest) consistently outperform both UniTS and SVM across all experimental conditions, including balanced and imbalanced datasets.
- On balanced datasets (e.g., IC1, Definition A), XGBoost achieved an accuracy of 0.77 compared to UniTS's 0.72.
- On large datasets (LGSR), XGBoost and RF reached accuracies of 0.94–0.96, significantly outperforming UniTS (0.80–0.87).
Robustness to Imbalance: Under class imbalance (1:4), tree-based models maintained higher recall and precision than UniTS and SVM. For instance, on the IC2 dataset, XGBoost maintained a recall of 0.57 under imbalance, whereas SVM's recall collapsed to 0.24.
Data Efficiency: A critical finding is the performance degradation of UniTS in data-scarce scenarios.
- On the full IC2 dataset, UniTS achieved 0.80 accuracy.
- On a subsampled version (IC2*), accuracy dropped to 0.73.
- On a further reduced version (IC2**), accuracy collapsed to 0.50.
- In contrast, XGBoost and RF maintained high accuracy (0.82–0.85) even with significantly reduced sample sizes, demonstrating superior data efficiency.
Churn Definition Impact: Performance generally improved with more lenient churn definitions (moving from A to C), likely due to increased sample sizes and easier class separability, but the relative ranking of models (Tree-based > UniTS > SVM) remained consistent.

Significance and Claims

The paper claims that for the specific task of churn prediction in non-contractual retail environments, conventional machine learning methods remain the superior choice over state-of-the-art transformer-based architectures like UniTS.

Practical Implications: The findings suggest that for typical retail applications where data is structured and often limited, traditional ensemble methods (XGBoost, Random Forest) offer a more reliable, efficient, and cost-effective solution. They achieve strong predictive performance with lower computational resource requirements and better generalization on smaller datasets.
Limitations of Modern AI: While UniTS is theoretically well-suited for modeling temporal dynamics, the study indicates it underperforms on small-to-medium tabular datasets due to high data requirements and complexity. The authors note that their results reflect the performance of UniTS under default configurations; specific fine-tuning for churn tasks might alter these outcomes, but the current evidence favors simpler models.
Future Directions: The paper suggests that future research should explore the integration of additional contextual features (e.g., browsing activity, promotions) and conduct systematic analyses of training time and energy efficiency to further assess the viability of advanced architectures in real-world deployment.

In conclusion, the study challenges the assumption that "bigger" or "more modern" models are always necessary, demonstrating that for retail churn prediction, the simplicity and data efficiency of traditional tree-based models provide a robust advantage.