🤖 machine learning

Interpretable vs Learned Encoders for High-Cardinality Fraud Detection

This study evaluates seven categorical encoding methods on the IEEE-CIS fraud dataset, finding that entity embeddings achieve the highest AUC-ROC (tied with CatBoost) by leveraging joint multi-column representations, while outperforming traditional tier group encoding and failing to match tree-based pipelines in AUC-PR.

Original authors: Xiao Han, Jingjing Liu, Moxuan Zheng, Zhen Zhang, Chenyu Wu

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiao Han, Jingjing Liu, Moxuan Zheng, Zhen Zhang, Chenyu Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a bank trying to spot a thief in a crowd of 590,000 people. Most of these people are honest, but about 3.5% are fraudsters. The problem is that the bank has a list of "clues" (like email addresses, device types, or card numbers), but many of these clues have thousands of unique variations. It's like trying to recognize a thief by their shoe size when there are 13,000 different shoe sizes in the crowd.

To solve this, the researchers had to figure out the best way to translate these messy, high-number clues into a format a computer can understand. They tested seven different "translation methods" (encoders) to see which one helped the computer spot the fraud best.

Here is the breakdown of their experiment and findings, using simple analogies:

The Setup: The "Translator" Contest

Think of the computer model (LightGBM) as a detective. The "encoders" are the translators who take the raw clues and explain them to the detective.

The researchers wanted to know: Does it matter who the translator is, or is the detective the only thing that matters? To find out, they kept the detective exactly the same and only changed the translator.

They tested seven translators:

The Dictionary (One-hot): Lists every single unique item. (Too long and clunky).
The Statistician (Target Encoding): Replaces a clue with the average "guilt score" of people who had that clue.
The Frequency Counter: Replaces a clue with how common it is.
The Grouping Manager (Tier Grouping): Sorts clues into buckets (e.g., "Low Risk," "Medium Risk," "High Risk") based on their guilt scores. This is designed to be easy for human auditors to read.
The Neural Network (Entity Embeddings): A smart system that learns a unique "fingerprint" for every clue by looking at how clues interact with each other.
The All-in-One Package (CatBoost): A pre-packaged system that does its own translation and detective work.
The Deep Learner (TabNet): A very complex, modern neural network.

The Results: Who Won?

1. The Smartest Translator (Entity Embeddings)
The Entity Embeddings method won the accuracy contest. It gave the detective the best view of the fraudsters (highest AUC-ROC score).

The Catch: It's like hiring a genius translator who speaks 30 different languages at once. It found patterns by looking at how clues worked together (e.g., a specific email domain combined with a specific device type). However, it is computationally expensive and harder for a human to explain why it made a decision.

2. The "Good Enough" & Auditable Translator (Tier Grouping)
The Tier Grouping method came in second (very close to the winner).

The Analogy: Imagine sorting all the clues into 5 clear buckets: "Very Safe," "Safe," "Okay," "Risky," and "Very Risky."
Why it matters: If a bank auditor asks, "Why did you flag this person?" the answer is simple: "They were in the 'Risky' bucket." It is fast, cheap, and easy to explain. It only lost a tiny bit of accuracy compared to the genius method.

3. The Heavyweight Champion (CatBoost)
The CatBoost system (which is a different type of detective entirely) actually won on a different metric (AUC-PR), which is better for spotting the rare fraudsters in a huge crowd. It was a statistical tie with the Entity Embeddings on the main metric.

4. The Disappointment (TabNet)
The TabNet model, which is a popular "deep learning" tool, failed to beat the simpler tree-based methods.

The Analogy: It was like bringing a super-complex robot to a job that a simple flashlight could do. When the data was scarce (less information available), the robot completely collapsed and performed poorly. The researchers found that using a standard, off-the-shelf TabNet didn't help here.

The Big Trade-Offs

The paper highlights three main things to consider when choosing a method:

Accuracy vs. Cost: The "Genius Translator" (Embeddings) was the most accurate, but it took 4 times longer to run than the "Grouping Manager" (Tier Grouping).
Accuracy vs. Explainability: In banking, you often have to explain your decisions to regulators (like the SR 11-7 rule). The "Genius Translator" is a "black box"—you can't easily explain its logic. The "Grouping Manager" is transparent; you can show the auditor exactly which bucket a person fell into.
The Metric Problem: Depending on which scorecard you use, the winner changes. If you care about the overall ranking, the Embeddings win. If you care about catching the rare fraudsters specifically, the CatBoost system wins.

The Bottom Line

The researchers concluded that there is no single "perfect" method.

If you want maximum accuracy and have the computing power, use Entity Embeddings.
If you need to explain your decisions to auditors and want speed, use Tier Grouping. It was almost as good as the best method but much easier to understand.
Don't assume that the most complex, modern deep-learning models (like TabNet) will always win; sometimes, a well-tuned, simpler approach works better.

In short: You don't always need a super-complex AI to catch a fraudster. Sometimes, a smart, organized filing system (Tier Grouping) is just as effective and much easier to defend in court.

Technical Summary: Interpretable versus Learned Encoders for High-Cardinality Fraud Detection

Problem Statement

The paper addresses the challenge of representing high-cardinality categorical variables (e.g., card identifiers, email domains, device fingerprints) in tabular fraud detection datasets. While classical machine learning benchmarks often focus on encoder performance within traditional model families, and deep learning studies often fix encodings to vary model architectures, there is a lack of controlled evidence comparing encoders across paradigms. This gap is critical for regulated finance (e.g., under SR 11-7), where model transparency and auditability are required alongside high accuracy. The authors seek to determine the trade-offs between accuracy, computational cost, and interpretability when selecting an encoding strategy for high-cardinality fraud data, specifically isolating the encoder's contribution from the downstream learner.

Methodology

The study utilizes the IEEE-CIS Fraud Detection dataset (590,540 records, 3.5% positive class, 8 high-cardinality columns with cardinalities ranging from 4 to 13,553).

Experimental Design

The authors conducted a controlled experiment using stratified 5-fold cross-validation with 3 repetitions (15 total runs). The core innovation is fixing the downstream learner for most comparisons to isolate the effect of the encoding method.

Seven Encoders Evaluated:

E1 (One-hot): Standard indicator encoding with frequency-based dropping.
E2 (Target Encoding): Out-of-fold mean target with Bayesian smoothing ( $m=30$ ).
E3 (Frequency): Mapping values to their relative frequency.
E4 (Tier Grouping): An interpretable method where Bayesian-smoothed fraud rates are binned into $K$ ordinal tiers ( $K \in \{3, 5, 7\}$ ). This produces auditor-readable boundaries.
E5 (CatBoost): Used as a cross-paradigm reference, consuming raw categoricals natively via permutation-based ordered target statistics.
E6 (Entity Embeddings $\to$ LightGBM): A shallow neural network learns dense embeddings for high-cardinality columns, which are then extracted and fed into a frozen LightGBM learner.
E7 (Target Encoding $\to$ TabNet): Uses the same target-encoded input as E2 but feeds it into the TabNet deep learning architecture.

Controlled Comparisons:

E1, E2, E3, E4, E6: All fed into the same frozen LightGBM learner (fixed hyperparameters).
E5 & E7: Serve as cross-paradigm references using different learners (CatBoost and TabNet, respectively).

Metrics & Statistics:

Primary Metric: AUC-ROC (fixed a priori to prevent metric shopping).
Secondary Metric: AUC-PR (for imbalance sensitivity).
Statistical Inference: Nadeau–Bengio corrected resampled t-test (to handle correlated CV folds) and Friedman/Nemenyi tests for ranking.

Key Results

1. Accuracy and Statistical Significance

Entity Embeddings (E6) achieved the highest AUC-ROC (0.9612), statistically tied with CatBoost (E5) at 0.9602.
Tier Grouping (E4) scored 0.9548, statistically significantly lower than E6 (+0.0064, $p < 0.001$ ) but statistically tied with TabNet (E7) and significantly higher than Frequency encoding.
Target Encoding (E2) (0.9571) outperformed Tier Grouping by a small but significant margin (0.0023, $p=0.0003$ ).
TabNet (E7) (0.9522) did not outperform the tree-based pipelines and performed comparably to Tier Grouping.

2. Metric Divergence (AUC-ROC vs. AUC-PR)

The study highlights a critical divergence in metric rankings:

AUC-ROC: Led by E6 (Embeddings) and E5 (CatBoost).
AUC-PR: CatBoost (E5) leads significantly (0.8216) over E6 (0.7928). The gap in AUC-PR (0.029) is an order of magnitude larger than the gap in AUC-ROC (0.001).
Conclusion: No single encoder dominates both metrics. The choice depends on deployment objectives (ranking quality vs. precision-recall balance).

3. Computational Cost and Efficiency

Tier Grouping (E4) is the most efficient, requiring 24.5 seconds/fold.
Entity Embeddings (E6) require 104.6 seconds/fold (4.3x E4) for a marginal AUC-ROC gain of 0.0064.
TabNet (E7) is the most expensive at 1654.3 seconds/fold (67x E4) and offers no accuracy advantage over tree-based methods in this setup.

4. Robustness and Sensitivity

Data Scarcity: Under subsampling (down to 10% data), TabNet (E7) collapsed (AUC-ROC dropped to 0.8108 with high variance), whereas tree-routed encoders (E1–E6) showed parallel, stable degradation.
Per-Column Analysis: When trained on single high-cardinality columns, Entity Embeddings (E6) underperformed classical encoders. This confirms that E6's advantage arises from joint multi-column representation (capturing latent co-occurrences across columns like card1 and DeviceInfo) rather than superior single-column encoding.
Tier Sensitivity: E4's performance was largely insensitive to the number of tiers ( $K$ ) or binning rules, with variations < 0.0019 AUC-ROC.

Key Contributions

Cross-Paradigm Controlled Experiment: The authors isolated the encoder as the controlled variable across both classical (LightGBM) and deep learning (TabNet) paradigms, attributing approximately 0.01 AUC-ROC variance solely to encoding choices.
Operationalized Auditor-Readable Encoding: The paper benchmarks "Tier Grouping," a method that bins smoothed fraud rates into ordinal tiers, demonstrating it is a competitive, low-cost, and fully auditable alternative to learned embeddings.
Interpretability-First Framing: The study explicitly scores interpretability and compute alongside accuracy, providing a framework for regulated finance where model-risk owners must justify scores (e.g., under SR 11-7).

Significance and Claims

The paper claims that encoder choice is an independent degree of freedom often confounded in previous studies that varied model families. The findings suggest:

Learned embeddings (E6) offer the highest AUC-ROC but at a higher computational cost and with reduced interpretability (30-dimensional vectors vs. clear tier boundaries).
Tier Grouping (E4) offers a "sweet spot" for regulated environments: it trails the top performer by only 0.0064 AUC-ROC but provides transparent, auditor-friendly boundaries and is significantly faster.
Off-the-shelf TabNet does not outperform tree-based pipelines on this task and is highly sensitive to data scarcity.
Metric Selection Matters: In fraud detection, AUC-ROC and AUC-PR can yield different "winners" (CatBoost wins on PR, Embeddings on ROC), necessitating that the choice of metric align with specific deployment goals.

The authors conclude that while learned embeddings provide a marginal accuracy edge through joint representation, tier grouping is a robust, interpretable, and computationally efficient alternative that satisfies strict regulatory requirements without significant performance loss.