On the Impact of Class Imbalance on the Learning Dynamics of Deep Neural Networks:An Intuitive Insight

Technical Summary: On the Impact of Class Imbalance on the Learning Dynamics of Deep Neural Networks

Problem Statement
Despite the widespread adoption of Deep Neural Networks (DNNs) in domains such as computer vision and natural language processing, these models struggle significantly with class imbalance, where one class (minority) has substantially fewer samples than another (majority). While classical machine learning literature often attributes poor minority performance to underfitting due to insufficient data, recent DNN-specific studies offer conflicting explanations. Some argue that DNNs underfit minority classes, while others claim they overfit minority samples, leading to non-generalizable representations due to feature deviation or biased decision boundaries. The core problem addressed by this study is the lack of a unified, intuitive understanding of how and why class imbalance disrupts the learning dynamics of DNNs, specifically regarding the temporal evolution of learning for majority versus minority classes.

Methodology
To systematically investigate these learning dynamics, the authors employed a controlled experimental approach:

• Datasets: The study utilized 30 binary imbalanced benchmark datasets from UCI and KEEL repositories, varying in Imbalance Ratio (IR) and class separability (measured by Silhouette Coefficient). Additionally, three balanced binary datasets were derived from the Iris dataset to serve as a baseline for comparison.
• Model Architecture: A fully connected DNN architecture using ReLU activation functions was used. The models incorporated Batch Normalization and a 0.5 dropout rate to mitigate overfitting.
• Optimization: Hyperparameters (depth and width) were optimized via 5-fold cross-validation to ensure models could achieve near-zero loss. Training utilized the Adam optimizer with cross-entropy loss, running for a minimum of 250 epochs.
• Evaluation Metrics: Beyond overall loss and accuracy, the study focused on class-specific metrics: True Positive Rate (TPR) for the minority class and True Negative Rate (TNR) for the majority class. The authors monitored the trajectory of per-class training losses and accuracies across epochs to observe the sequence of learning. Random oversampling was also applied as a baseline intervention to compare against the raw imbalance scenario.

Key Contributions and Findings
The study provides an intuitive insight into the learning trajectory of DNNs on imbalanced data, revealing a distinct temporal pattern:

1. Sequential Learning Dynamics: Unlike balanced datasets where DNNs learn all classes simultaneously with similar loss reduction rates, imbalanced datasets exhibit a sequential learning pattern. The model initially learns the majority class almost exclusively, driving the majority class loss down rapidly while the minority class loss often increases or remains stagnant in early epochs.
2. Early Underfitting of Minority Classes: During the initial training phase, the DNN effectively underfits the minority class, classifying nearly all samples (including minority ones) as the majority class. This results in a TPR of 0% in early epochs.
3. Overfitting for Loss Minimization: Although the overparameterized nature of DNNs eventually allows them to learn the minority class samples (reducing minority loss to near zero), this learning occurs only after the majority class is mastered. The authors posit that the model learns the minority representations not to generalize well, but merely to minimize the overall training loss. Consequently, these learned minority representations are non-generalizable at the test phase, resulting in low TPR despite high training accuracy.
4. Role of Data Complexity and Sample Size: The severity of this phenomenon is modulated by the Imbalance Ratio, data complexity (overlap), and minority sample size.
   • Datasets with well-separated clusters (high Silhouette Coefficient) do not suffer from the early underfitting phase, even with high IR.
   • Datasets with low IR but very small minority sample sizes (e.g., only 7 minority samples) suffer severe early underfitting.
   • The overall training loss trajectory is dominated by the majority class, masking the poor performance on the minority class during early training.
5. Limitations of Oversampling: Applying random oversampling improved minority recognition (TPR) but often at the expense of majority class performance (TNR), and significant gaps remained for highly imbalanced datasets.

Significance and Claims
The paper claims that its primary significance lies in clarifying the mechanism behind DNN failure on imbalanced data. It challenges the binary debate of "underfitting vs. overfitting" by demonstrating that DNNs experience both: initial underfitting of the minority class followed by a form of overfitting where minority samples are memorized to satisfy global loss minimization without achieving generalization.

The authors conclude that the poor performance of DNNs on minority classes is not merely a result of data scarcity but a specific learning dynamic where the model prioritizes the majority class to drive down total loss. This insight suggests that effective imbalance methods must address the temporal aspect of learning dynamics, rather than just static data distribution. The study modestly asserts that while DNNs can fit imbalanced data to near-zero loss given sufficient time, the resulting decision boundaries are biased, and the model's ability to generalize to minority test samples is severely compromised unless the class clusters are well-separated.