📊 statistics

Challenges in the calibration of tree-based models for imbalanced classification

This paper demonstrates that analytically calibrating tree-based models trained on undersampled imbalanced data leads to unreliable prevalence estimates and unexpected biases, arguing instead for the use of learning-based calibration methods like beta calibration.

Original authors: Nathan Phelps, Daniel J. Lizotte, Douglas G. Woolford

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nathan Phelps, Daniel J. Lizotte, Douglas G. Woolford

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Recipe" Problem

Imagine you are a chef trying to learn how to make a perfect soup. However, your kitchen is incredibly unbalanced: you have 10,000 potatoes but only 10 carrots. If you try to learn the recipe by tasting a bowl that has 10,000 potatoes and 10 carrots, you will never learn what a carrot tastes like. You'll think the soup is 99.9% potato.

To fix this, you decide to undersample. You throw away most of the potatoes so you have a bowl with 50 potatoes and 10 carrots. Now, your training data is balanced, and you can learn the recipe better.

The Catch: When you finally cook the soup for the real world (where there are actually 10,000 potatoes), your taste buds are still used to the balanced bowl. You think the soup is 50% carrots, but in reality, it's still 99.9% potato. Your predictions are "biased."

The Old Fix: The "Math Formula"

For a long time, data scientists used a specific math formula (Equation 1 in the paper) to fix this. The idea was simple: "We know we threw away 90% of the potatoes. Let's just multiply our prediction by a number to mathematically 'add them back'."

The authors of this paper say: "Stop doing that."

They found that this math formula doesn't work well for Tree-Based Models (like Random Forests). Instead of fixing the soup, the formula makes the taste wildly unpredictable.

The Two Main Problems Discovered

1. The "Knob" Problem (Number of Predictors)

Imagine your Random Forest is a team of 500 detectives trying to solve a mystery.

The Rule: At every step, the detectives can only look at a certain number of clues (predictors) to decide which way to go.
The Experiment: The authors turned a "clue knob." Sometimes they let the detectives look at just 2 clues. Sometimes they let them look at 15 clues.

What happened?
When they used the old math formula to fix the bias, the final answer changed drastically depending on how many clues the detectives were allowed to look at.

If they looked at 2 clues, the model predicted 100 fires.
If they looked at 15 clues, the model predicted 140 fires.

The Analogy: It's like asking a group of people to guess the weight of a watermelon. If you tell them to only look at the color, they guess 10 lbs. If you tell them to look at the color, the stem, the sound when tapped, and the texture, they guess 15 lbs.
In a perfect world, the math formula should have corrected these guesses so they all landed on the same true weight. But instead, the formula made the guesses drift further apart. The more clues the detectives used, the more the final estimate jumped up.

2. The "Surprising Bias" (Sampling Rate)

Usually, we think that if you have too few "minority" items (like the 10 carrots), the model will ignore them and guess too low.

The Old Belief: "Decision trees ignore the minority class."
The Paper's Discovery: In their experiments, the decision trees actually did the opposite! They overestimated the minority class.

The Analogy: Imagine you are trying to guess how many red marbles are in a jar of 1,000 marbles (990 blue, 10 red). You take a small sample.

Expected: You might guess "maybe 1 or 2 red marbles" (underestimating).
What the Trees Did: The trees looked at the sample and said, "Wow, red marbles are everywhere! I bet there are 50 red marbles!"

The authors found that as they changed the sampling rate (how many potatoes they threw away), the model's overestimation got worse. The math formula tried to fix this, but because the tree was already guessing too high, the formula just made the final result even more chaotic.

The Real-World Example: Wildfires

The authors tested this with real data about wildfires in Alberta, Canada.

Wildfires are rare (like the carrots).
They used the "math formula" to fix the models.
The Result: Depending on how they set the "clue knob" (number of predictors) or how they sampled the data, the model predicted the number of future fires could vary by 40%.
Why this matters: If a government agency uses these models to decide where to send fire trucks, they might send too many trucks to one area and none to another, simply because they tweaked a setting in the computer code.

The Conclusion

The paper concludes that the "math formula" (analytical calibration) is a broken tool for this specific job. It creates a situation where:

Changing a setting (like the number of clues) changes the answer, even though the answer shouldn't change.
The models sometimes guess the minority class is more common than it actually is, which the formula fails to fix.

The Solution: Instead of using a rigid math formula, the authors suggest using "learning" methods (like Beta Calibration). Think of this as hiring a taste-tester who actually tastes the final soup and learns how to adjust the seasoning, rather than trying to calculate the adjustment with a calculator.

Summary in One Sentence

Using a simple math formula to fix "unbalanced" data in tree-based models doesn't work; it makes the predictions wildly unstable and sometimes causes the model to overestimate rare events, so we need smarter, learning-based ways to fix the bias instead.

Technical Summary: Challenges in the Calibration of Tree-Based Models for Imbalanced Classification

Problem Statement
In imbalanced binary classification, it is common practice to undersample the majority class to create a balanced training dataset. However, this procedure biases the model's predicted probabilities because the training distribution no longer represents the underlying population. To correct this, analytical calibration methods (specifically Eq. 1 derived by Dal Pozzolo et al., 2015) have been proposed to map predictions from undersampled models back to the true population probabilities based on the sampling rate. While widely applied to models like Random Forests (RF), this paper investigates whether this analytical approach yields well-calibrated probability estimates when applied to tree-based models trained on undersampled data.

Methodology
The authors employ a multi-faceted approach to evaluate the efficacy of analytical calibration:

Theoretical Analysis: The paper first examines the mathematical validity of applying Eq. 1 to Balanced Random Forests (Chen et al., 2004). It demonstrates that the derivation of Eq. 1 relies on assumptions regarding sampling probabilities that are violated by the bootstrapping procedure inherent in Balanced Random Forests.
Empirical Case Study: Using a real-world dataset of human-caused wildfires in the Lac La Biche region of Alberta (1996–2016), the authors train standard Random Forests on undersampled data. They systematically vary two hyperparameters: the sampling rate of the majority class and the number of predictors considered at each split ( $m_{try}$ ). The models are calibrated using Eq. 1, and the resulting prevalence estimates (sum of predicted probabilities) are compared against the true number of events.
Simulation Study: To isolate variables and control the data-generating process, a simulator is developed with 10 predictors and a complex non-linear relationship (Eq. 2). This allows for the generation of datasets with known true probabilities. The authors repeat the empirical case study on these simulated datasets to assess the consistency of the findings.
Mechanism Investigation:
- Effect of $m_{try}$ : The authors analyze the distribution of predictions before and after calibration using quantile-quantile plots to understand how the non-linearity of Eq. 1 interacts with the "extremeness" of Random Forest predictions.
- Effect of Sampling Rate: The authors test individual Decision Trees (fit to purity) on datasets with varying levels of imbalance to determine if the trees themselves exhibit a bias toward the minority class, which would explain the systematic overestimation observed in the Random Forests.

Key Contributions
The paper makes three primary contributions:

Inapplicability to Balanced Random Forests: It demonstrates that analytical calibration (Eq. 1) is theoretically unsuitable for Balanced Random Forests because the bootstrapping process alters the probability of inclusion for minority class observations ( $\mathbb{P}(S=1|Y=1) \approx 0.632$ ), violating the assumptions required for the equation's derivation.
Hyperparameter Sensitivity in Standard Random Forests: It reveals that for standard Random Forests trained on undersampled data, analytical calibration leads to prevalence estimates that are not invariant to hyperparameter choices. Specifically, the estimated prevalence increases systematically with both the number of predictors considered at each split and the sampling rate used.
Discovery of Minority Class Bias in Decision Trees: Contradicting the prevailing literature which suggests decision trees are biased toward the majority class (ignoring or underestimating the minority), the authors provide evidence that decision trees can be biased toward the minority class (overestimating probabilities) when trained on imbalanced data. This bias persists even after analytical calibration, leading to systematic overprediction.

Results

Wildfire Prediction: In the real-world application, the predicted number of wildfires varied by more than 40% depending on the choice of hyperparameters, despite the models being calibrated. The estimates showed clear positive correlations with both the sampling rate and the number of predictors considered at each split.
Simulation: The simulation study confirmed the real-world findings, showing that prevalence estimates increased with the sampling rate and the number of predictors. In many cases, the calibrated models systematically overestimated the true prevalence by up to 20%.
Mechanism of $m_{try}$ : The analysis showed that Random Forests produce more extreme predictions (closer to 0 or 1) when more predictors are considered at each split. While the pre-calibration distributions were similar, the non-linear nature of Eq. 1 amplified these differences post-calibration, causing the model with higher $m_{try}$ to yield significantly higher prevalence estimates.
Mechanism of Sampling Rate: The investigation into individual decision trees revealed that as class imbalance increased, the ratio of predicted probability to true probability increased (exceeding 1.0), indicating an overprediction bias toward the minority class. This bias was not eliminated by analytical calibration.

Significance and Claims
The authors conclude that analytical calibration is unsuitable for calibrating tree-based models trained on undersampled data. The method fails to produce well-calibrated probability estimates because the resulting prevalence depends on arbitrary hyperparameter choices and inherent biases in the tree structure that the analytical formula cannot correct.

The paper argues that because good prevalence estimates are a necessary condition for good calibration, the failure to produce stable prevalence estimates indicates a fundamental flaw in the approach. Consequently, the authors suggest that tree-based models trained on undersampled data should utilize calibration methods capable of learning the miscalibration pattern from the data, such as beta calibration or modified Platt's scaling, rather than relying on analytical corrections.

The finding that decision trees can be biased toward the minority class is presented as a significant counterpoint to existing literature, suggesting that practitioners should not automatically assume decision trees will neglect the minority class in imbalanced settings. However, the authors remain modest, noting that this bias was not observed consistently across all configurations (e.g., in some R implementation variations) and that further work is needed to fully understand the conditions under which this bias occurs.