🤖 machine learning

An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification

This paper investigates how balancing methods in imbalanced classification inflate predictive multiplicity (the Rashomon effect) by introducing a new metric called "obscurity" and proposing an extended performance-gain plot to responsibly monitor the trade-off between prediction accuracy and model selection uncertainty.

Original authors: Mustafa Cavus, Przemysław Biecek

Published 2026-05-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mustafa Cavus, Przemysław Biecek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Many Right Answers" Problem

Imagine you are trying to predict who will win a game of chess. You have a dataset of past games, but there's a problem: 90% of the games were won by White, and only 10% by Black. This is called an imbalanced dataset.

If you train a computer model on this data without fixing it, the computer gets lazy. It learns that "White always wins," so it predicts "White" for every single new game. It gets 90% accuracy, but it fails completely at predicting the Black wins.

To fix this, data scientists use balancing methods. They act like a chef adding ingredients to a soup:

Oversampling: They copy the rare "Black win" games to make more of them (like adding more salt).
Undersampling: They throw away some of the common "White win" games to make the pot smaller (like removing excess water).

The goal is to make the computer pay attention to the minority class. But this paper asks a scary question: Does fixing the data create a new kind of confusion?

The Rashomon Effect: The "Many Truths" Phenomenon

The paper uses a concept called the Rashomon Effect. Imagine a crime scene where five different witnesses give five different stories. All five stories are plausible and fit the facts equally well, but they contradict each other.

In machine learning, the Rashomon Set is a group of different computer models that all perform almost exactly the same (they are all "plausible"), but they make different predictions for the same person or situation.

Model A says: "This loan applicant is safe."
Model B says: "This loan applicant is risky."
Both models have the same overall accuracy score.

If you just pick one model randomly (blind selection), you might accidentally pick the one that denies a loan to a good person, even though another equally accurate model would have approved it. This is called Predictive Multiplicity.

The Experiment: What Happens When We Balance the Data?

The authors wanted to know: Does using balancing methods (oversampling/undersampling) make this "Many Truths" problem worse?

They took 21 different real-world datasets (like spam detection, credit card fraud, and wine quality) and ran them through a "model factory" (a tool called Forester) that creates hundreds of slightly different models for each dataset. They did this twice:

Using the original, unbalanced data.
Using balanced data (after applying the balancing methods).

They then measured three things to see how much the models disagreed with each other:

Ambiguity: How many people get conflicting answers? (e.g., "5 out of 100 people get different predictions.")
Discrepancy: What is the worst-case scenario? (e.g., "One specific model disagrees with the others on 20 people.")
Obscurity (New Metric): This is the paper's new idea. It measures the average amount of confusion. Instead of just asking "Is there a conflict?", it asks "How messy is the conflict on average?" Think of it as measuring the "fog" over the data.

The Findings: The "Fix" Made the Fog Thicker

Here is what they discovered:

Balancing Methods Increase Confusion: When they used balancing methods (like SMOTE or random oversampling), the Obscurity and Discrepancy went up.
- The Analogy: Imagine you are trying to find a lost dog in a park. In the original park (unbalanced data), all the search dogs agree on where the dog is. But when you add "balancing" (like adding more search dogs or changing the terrain), the search dogs start barking in different directions. They are all still "good" dogs, but they can't agree on the location.
The "Partial" Fix Didn't Work: Some experts suggested using "partial resampling" (only balancing the data a little bit, not 100%) to avoid this mess. The authors tested this, but found it didn't help. The confusion remained high regardless of how much they balanced the data.
Variable Importance Changed: They also checked if the models were looking at different clues. For example, one model might think "income" is the most important factor, while another thinks "age" is. They found that while the order of importance didn't change drastically in a statistical sense, the balancing methods did make the models behave differently overall.

The Solution: A New Dashboard

Since they couldn't stop the confusion from happening, the authors proposed a way to monitor it.

They suggested using an Extended Performance-Gain Plot.

Imagine a dashboard with two dials.
Dial 1 (Horizontal): Shows how much better the model gets at predicting correctly (Performance Gain).
Dial 2 (Vertical): Shows how much the models disagree with each other (Multiplicity/Obscurity).

The Lesson: You shouldn't just look at Dial 1. You might find a method that improves accuracy slightly (Dial 1 goes up), but it causes a massive explosion in disagreement (Dial 2 goes way up). The authors say you need to look at both dials to make a responsible decision.

Summary of the Paper's Claims

Balancing data is necessary for imbalanced problems, but it has a hidden cost.
Balancing methods increase "Predictive Multiplicity." They create a situation where many models are equally accurate but give conflicting answers to the same people.
New Metric: They introduced "Obscurity" to measure how "foggy" or conflicting these model predictions are on average.
Partial Resampling is not a magic bullet. It doesn't solve the problem of increased confusion.
Responsible AI: When choosing a model, you must check not just if it is accurate, but if it is stable. If you pick a model blindly from a "Rashomon Set" created by balancing methods, you risk making arbitrary decisions that hurt individuals.

The paper concludes that while balancing methods are a "haven" for solving imbalanced data, researchers must be careful not to walk blindly into a fog of conflicting predictions. They need to use the new "dashboard" (the extended plot) to see the trade-off between accuracy and stability.

Technical Summary: An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification

Problem Statement
In data-centric AI, class imbalance is a prevalent challenge where models tend to favor the majority class, resulting in poor performance on the minority class. While balancing methods (e.g., oversampling, undersampling) are standard data-centric approaches to mitigate this, their impact on model behavior remains under-explored. Specifically, there is a lack of understanding regarding how these methods influence the Rashomon effect—a phenomenon where multiple models achieve approximately equal predictive accuracy but may yield conflicting predictions for the same observations. This leads to predictive multiplicity, where blind model selection from a set of equally accurate models can result in conflicting outcomes, posing risks to model validation, explanation, and fairness. The paper investigates whether balancing methods inflate this predictive multiplicity and alter variable importance orders.

Methodology
The authors conducted experiments using a dataset of 21 imbalanced benchmark datasets with imbalance ratios ranging from 1.54 to 129.53. The study employed the following methodological framework:

Balancing Methods: Four techniques were evaluated: Random Oversampling, SMOTE (Synthetic Minority Over-sampling Technique), Random Undersampling, and Near Miss.
Rashomon Set Construction: The Forester (a tree-based AutoML tool) was utilized to generate the Rashomon set. This approach allows for the control of the set size via Bayesian optimization parameters, creating an empirical hypothesis space of models with loss values within a specific threshold ( $\epsilon = 0.05$ ) of the reference model.
Metrics for Predictive Multiplicity:
- Existing Metrics: Ambiguity (ratio of observations with conflicting predictions) and Discrepancy (maximum ratio of conflicting observations between the reference and any competing model).
- Proposed Metric: Obscurity, defined as the average ratio of conflicting predictions for observations across all models in the Rashomon set. This metric aims to capture the abundance of observation conflicts more granularly than binary ambiguity.
- Variable Importance Order Discrepancy (VIOD): A new metric proposed to measure the dissimilarity in variable importance orders between the reference model and other models in the set, based on Kendall's $\tau$ correlation coefficient.
Experimental Variables: The study varied resampling ratios (resulting imbalance ratios of 1.0 to 1.25) to test partial resampling as a potential mitigation strategy.
Statistical Analysis: Results were summarized using statistical plots and evaluated via Kruskal-Wallis, Friedman, and Dunnett's pairwise tests.

Key Results
The experiments addressed four specific research questions (RQs):

RQ1 (Impact on Predictive Multiplicity): Balancing methods were found to inflate predictive multiplicity. Visual analysis and statistical testing (Kruskal-Wallis, $p < 0.05$ ) demonstrated that models trained on balanced datasets exhibited significantly higher obscurity and discrepancy compared to those trained on original, imbalanced data. The zones of balancing methods on the obscurity axis did not overlap with the original dataset zones, indicating a distinct increase in conflicting predictions.
RQ2 (Impact on Variable Importance): Regarding the Variable Importance Order Discrepancy (VIOD), the study found no statistically significant difference in the median VIOD values between models trained on balanced versus original datasets. This contrasts with some previous findings on smaller datasets, suggesting that while prediction conflicts increase, the fundamental ranking of variable importance remains relatively stable across the Rashomon set in this broader experimental context.
RQ3 (Partial Resampling): The hypothesis that partial resampling (e.g., targeting an imbalance ratio of 1.25) could mitigate model behavior changes was rejected. Statistical tests (Friedman test) showed no significant difference in Rashomon metrics across varying resampling ratios, indicating that partial resampling does not solve the multiplicity problem introduced by balancing.
RQ4 (Monitoring Trade-offs): The study validated the utility of an extended performance-gain plot. This visualization maps the trade-off between performance gain (measured by AUC improvement) and multiplicity (obscurity/discrepancy). The results highlighted that while oversampling methods (Oversampling, SMOTE) generally improved performance, they also increased multiplicity. Conversely, undersampling methods sometimes failed to improve performance while still increasing multiplicity.

Significance and Claims
The paper claims to be the first study to examine the Rashomon effect specifically in the context of balancing methods for imbalanced classification. Its primary contributions are:

Empirical Evidence of Inflation: It provides evidence that balancing methods, while intended to improve accuracy, inadvertently increase predictive multiplicity (conflicting predictions) within the Rashomon set.
New Metric (Obscurity): The introduction of obscurity offers a more nuanced view of predictive multiplicity by measuring the average ratio of conflicts rather than just the existence of conflict.
Refutation of Partial Resampling: It challenges the notion that partial resampling is a sufficient solution for balancing-induced model behavior changes.
Responsible Modeling Framework: The authors propose using the extended performance-gain plot as a practical tool for data-centric AI practitioners. This tool allows for the monitoring of the trade-off between performance gains and the risks associated with predictive multiplicity, advocating for a more responsible model selection process that considers not just accuracy, but also the stability and consistency of model predictions.

The paper concludes that while balancing methods are a "haven" for imbalanced classification, researchers must account for the increased risk of multiplicity. Blind selection from the resulting Rashomon set is risky; therefore, the extended performance-gain plot is essential for conducting modeling processes responsibly.