Technical Summary: DAMEL – Dual-Axis Multi-Expert Learning for Class-Imbalanced Learning

1. Problem Statement

Deep neural networks (DNNs) often struggle with class-imbalanced learning (CIL) found in real-world long-tailed distributions. While existing CIL techniques (e.g., resampling, reweighting, decoupled learning) successfully reduce prediction bias toward majority classes, they frequently introduce a trade-off: an increase in prediction variance.

Recent multi-expert learning algorithms attempt to mitigate this variance by training multiple experts. However, these methods typically rely on complex, multi-stage procedures involving knowledge distillation or specific data splitting strategies. Furthermore, existing multi-expert approaches generally aggregate the predictions of multiple experts, which may not fully leverage the diverse feature representations learned by each expert.

2. Methodology: DAMEL

The authors propose DAMEL (Dual-Axis Multi-Expert Learning), a conceptually simple, one-stage algorithm that reduces both squared bias and prediction variance by leveraging multiple experts along two distinct axes: the representation axis and the time axis.

Architecture and Training

• Shared Backbone: DAMEL utilizes $K$ experts that share a deep CNN backbone (up to the last residual block) to minimize computational costs. Each expert possesses an individual last residual block and a classification layer.
• Representation-Axis Ensemble:
  • Instead of averaging representations or aggregating final predictions, DAMEL concatenates the representation vectors ($z_k$) from all $K$ experts.
  • An auxiliary balanced classifier is trained on these concatenated representations using a class-balanced loss (which reweights the cross-entropy loss inversely proportional to class frequency).
  • Crucially, the gradients from the class-balanced loss are backpropagated only to the auxiliary classifier, while the backbone and individual expert heads are trained via standard cross-entropy loss. This allows the auxiliary classifier to exploit complementary features from all experts without forcing the backbone to learn a single, potentially compromised representation.
  • The entire system (experts + auxiliary classifier) is trained end-to-end, avoiding the decoupled training trends seen in other CIL methods.

Time-Axis Ensemble

• To further reduce variance, DAMEL aggregates network weights across training epochs using an Exponential Moving Average (EMA).
• Epoch-Level Update: Unlike conventional iteration-level EMA or Stochastic Weight Averaging (SWA), DAMEL updates the EMA weights ($\theta_{EMA}$) only at the end of each epoch. This strategy increases the diversity of the aggregated weights by ensuring that consecutive updates are less correlated, capturing more distinct model states.
• Inference: During the test phase, predictions are made solely using the auxiliary classifier with the aggregated $\theta_{EMA}$ weights. The individual expert predictions are not used for the final output.

3. Key Contributions

The paper highlights five primary contributions:

1. Representation-Axis Ensemble: A novel mechanism that aggregates expert representations via concatenation rather than prediction averaging, enabling an auxiliary classifier to utilize diverse, complementary features from multiple experts.
2. Preservation of Diversity: Demonstration that representation concatenation is not merely a dimensionality increase but a mechanism to preserve expert-specific representation diversity and jointly utilize discriminative cues.
3. Time-Axis Ensemble: The introduction of an epoch-level EMA weight aggregation strategy, distinct from iteration-level EMA and SWA, which treats temporally aggregated weights as an additional ensemble axis to reduce prediction variance.
4. End-to-End Auxiliary Classifier: The design of an auxiliary balanced classifier trained on concatenated representations with a class-balanced loss, reducing bias toward majority classes while leveraging diverse features in a single training stage.
5. Empirical Validation: Extensive experiments and ablation studies demonstrating that the dual-axis approach jointly reduces squared bias and prediction variance, outperforming existing multi-expert and CIL algorithms.

4. Experimental Results

DAMEL was evaluated on four benchmark datasets: CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and iNaturalist2018.

• Performance: DAMEL achieved state-of-the-art performance across all datasets and imbalance ratios ($\gamma = 10, 50, 100$). For instance, on CIFAR-100-LT ($\gamma=100$), DAMEL with 4 experts achieved 53.1% overall accuracy, outperforming strong baselines like RIDE (49.1%), ACE (49.6%), and TLC (49.8%).
• Bias-Variance Trade-off: Quantitative analysis of squared bias and variance showed that DAMEL successfully reduced both metrics simultaneously. While rebalancing techniques reduced bias but increased variance, and multi-expert methods reduced variance, DAMEL's dual-axis approach lowered both.
• Ablation Studies:
  • Removing the EMA weights or the representation concatenation significantly degraded performance.
  • Concatenating representations outperformed aggregating predictions.
  • Epoch-level EMA outperformed iteration-level EMA and simple SWA, confirming the importance of update frequency in constructing an effective time-axis ensemble.
  • Capacity-controlled studies proved that performance gains stem from expert diversity rather than simply increased model capacity or feature dimensionality.
• Robustness: The method remained effective when combined with strong data augmentation techniques (AutoAugment, Cutout) and showed stability across various hyperparameter settings (number of experts, EMA $\beta$, classifier scale).

5. Significance and Claims

The authors claim that DAMEL offers a conceptually simple yet highly effective solution to the bias-variance trade-off in class-imbalanced learning. By avoiding complex multi-stage distillation or data splitting, DAMEL achieves superior accuracy through a one-stage training process.

The paper emphasizes that the dual-axis strategy is the core innovation:

• The representation axis ensures the classifier has access to a rich, diverse set of features, reducing bias.
• The time axis (via epoch-level EMA) stabilizes the model and reduces variance by averaging weights from diverse optimization states.

The authors conclude that while their current hyperparameter selection is heuristic, the proposed framework provides a universal and robust approach for CIL, effectively mitigating the limitations of both traditional rebalancing techniques and existing multi-expert algorithms.