CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data

1. Problem Statement

The paper addresses the critical challenge of class imbalance in machine learning, specifically within the context of Large Language Models (LLMs) and Small Language Models (SLMs).

• The Core Issue: Traditional ensemble methods (e.g., majority voting, confidence-weighted averaging) inherently favor majority classes. This leads to the marginalization of minority classes, resulting in poor performance on macro-level metrics (like Macro F1-score) and unfair outcomes in domains where minority classes are statistically rare but pedagogically or socially significant (e.g., "To some extent" in educational feedback or specific emotions like "surprise" and "love" in sentiment analysis).
• The Gap: Existing ensemble techniques lack mechanisms to dynamically detect and boost minority signals without treating them as noise. Current approaches often sacrifice minority performance for overall accuracy, and reasoning-augmented methods remain vulnerable to imbalance.

2. Methodology: The CAMO Framework

The authors propose CAMO (Class-Aware Minority-Optimized), a novel, domain-agnostic ensemble technique designed to preserve and amplify minority class predictions through a hierarchical, multi-stage decision process.

A. Core Components

CAMO integrates three primary signals to make final predictions:

1. Vote Distribution: Counts of model votes for each class.
2. Confidence Calibration: Analysis of the confidence scores ($p_m$) assigned by individual models.
3. Inter-Model Uncertainty: Measurement of disagreement and dispersion among the ensemble members.

B. The Hierarchical Decision Pipeline

CAMO employs a 7-stage hierarchical review to determine the final class label. The process stops at the first satisfied condition, prioritizing minority preservation:

1. Unanimity: If all models agree ($V(c) = M$), that class is selected.
2. Strong Minority Consensus: Boosts minority classes if they meet specific thresholds for vote count ($\ge \theta$) and average confidence ($\bar{p}$), or if a single high-confidence vote exists.
3. Isolated High-Confidence Minority Vote: Prioritizes a single minority vote if it has very high confidence ($p_{max} > \tau$) while the overall ensemble confidence is low (indicating uncertainty).
4. Uncertainty-Triggered Prioritization: If the ensemble is uncertain (low average confidence or high variance), minority classes with at least one vote are prioritized.
5. Confidence-Weighted Aggregation with Boosting: Calculates a score $S(c)$ for each class. For minority classes ($c \in C_{min}$), a dynamic boost function $\beta$ is applied, increasing the score based on vote count and low average confidence.
6. Minority Vote Under Competitive Dominance: If a minority class has the highest vote count and a high-confidence vote, it overrides the majority.
7. Standard Majority Rule: If no previous conditions are met, the standard argmax of the aggregated scores is used.

C. Dynamic Boosting Function

The boost term $B_c(v, \bar{p})$ for minority classes is defined as:
$$B_c(v, \bar{p}) = \min \left( \beta_{base}^c + \sum_{i=1}^4 \alpha_i^c I[\text{condition}_i], \beta_{max}^c \right)$$
This function triggers increases based on conditions such as having $\ge 2$ or $\ge 3$ votes, or when average confidence drops below specific thresholds (e.g., 0.7 or 0.6), ensuring minority signals are amplified when the ensemble is unsure.

3. Experimental Setup

The authors validated CAMO on two highly imbalanced, domain-specific benchmarks using eight different language models (3 LLMs and 5 SLMs) under both zero-shot and fine-tuned settings.

• Datasets:
  1. BEA 2025 Mistake Identification Track: A ternary classification task where the minority class ("To some extent") comprises only ~7% of the data.
  2. DIAR-AI/Emotion Dataset: An emotion recognition task with significant long-tail distributions, specifically focusing on rare emotions like "surprise" (3%) and "love" (8%).
• Baselines: CAMO was compared against seven established ensemble algorithms, including Majority Voting, Confidence-Weighted Voting, Class-Balanced weighting, Uncertainty-Aware weighting, and Meta-Ensembles.
• Models Tested: Ranged from 0.6B (Qwen3) to 8B (Llama-3.1, DeepSeek-R1) parameters, utilizing LoRA for parameter-efficient fine-tuning.

4. Key Results

The experiments demonstrated that CAMO consistently outperforms state-of-the-art ensemble methods, particularly in Strict Macro F1-Score.

• BEA 2025 Results:

  • In fine-tuned settings, CAMO achieved the highest Strict Macro F1 across almost all models. For example, on the Phi-3 (3.8B) model, CAMO achieved 85.6% Strict F1, significantly outperforming the next best (MSA Base at 82.6%).
  • On smaller models like DeepSeek-R1 (1.5B), CAMO improved the F1 score from ~55% (standard baselines) to 57.6%, showing robustness even with limited model capacity.
  • In zero-shot settings, while absolute scores were lower, CAMO maintained superior performance in preserving minority class detection compared to baselines that often collapsed to majority class predictions.

• DIAR-AI/Emotion Results:

  • CAMO significantly improved performance on minority emotions. For the Llama-8B model, the F1 score for the minority class "surprise" increased from 0.72 (baseline) to 0.75, and "love" improved from 0.81 to 0.84.
  • The overall weighted F1 and Macro F1 scores were highest for CAMO, demonstrating that boosting minority classes did not degrade majority class performance.

• Ablation & Analysis:

  • The study confirmed that CAMO's efficacy is synergistic with model adaptation (fine-tuning).
  • The hierarchical decision structure successfully prevented minority classes from being drowned out by majority consensus in cases of model disagreement.

5. Key Contributions

1. Novel Ensemble Architecture: Introduction of CAMO, a hierarchical ensemble method that explicitly integrates vote distribution, confidence calibration, and uncertainty to protect minority classes.
2. Dynamic Minority Boosting: A mathematical formulation that dynamically increases the weight of minority predictions based on ensemble uncertainty and vote counts, rather than static re-weighting.
3. Comprehensive Benchmarking: The most extensive comparative evaluation to date for imbalanced classification, covering 8 models, 2 distinct domains (Education and Emotion), and both zero-shot and fine-tuned paradigms.
4. State-of-the-Art Performance: Establishment of new benchmarks for Macro F1-scores on imbalanced datasets, proving that ensemble design can be as critical as model selection.

6. Significance and Impact

• Fairness in AI: CAMO provides a framework for reducing bias in AI systems, which is crucial for sensitive applications like educational assessment (where "partial credit" or "to some extent" feedback is vital) and mental health/emotion recognition (where rare emotional states must be detected).
• Domain Agnosticism: The method is not tied to specific data types, making it applicable to any classification task with underrepresented but essential classes (e.g., fraud detection, rare disease diagnosis).
• Practical Deployment: The paper demonstrates that CAMO can be implemented as a post-processing layer (a posteriori) on existing models, offering a cost-effective way to improve fairness and robustness without requiring massive retraining of base models.

In conclusion, the paper argues that CAMO is a reliable, robust solution for the class imbalance problem, bridging the gap between high overall accuracy and fair performance across all classes in modern language model applications.