💬 NLP

Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss

This paper investigates the use of weighted loss functions in Transformer-based models for multi-label emotion detection on the BRIGHTER dataset, finding that while the approach effectively boosts performance on high-frequency classes, it offers limited improvement for minority emotion classes.

Original authors: Xia Cui

Published 2026-02-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xia Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read short messages and guess the emotions inside them, like "joy," "anger," or "sadness." This is tricky because a single message can have multiple emotions at once (like being "angry but also surprised").

The main problem the paper tackles is that the robot's training data is unbalanced. It's like trying to teach a student to recognize animals, but 90% of the pictures you show them are of dogs, and only a few are of tigers. Naturally, the student becomes an expert at spotting dogs but completely misses the tigers. In the world of text, this means the AI gets really good at spotting common emotions (like "joy") but fails to notice rare ones (like "fear").

Here is how the authors tried to fix this, using simple analogies:

The Problem: The "Loud Majority"

In the dataset used for this study (called BRIGHTER), some emotions appear thousands of times, while others appear only a handful of times. When the AI learns, it listens to the "loud" emotions (the majority) and ignores the "quiet" ones (the minority). Traditional ways to fix this involve physically copying the rare examples or deleting the common ones (like photocopying the few tiger pictures to make a bigger stack), but the authors found this messy and complicated for text.

The Solution: The "Volume Knob"

Instead of changing the stack of pictures, the authors tried a different approach: Weighted Loss.

Think of the AI's learning process like a teacher grading a student's homework.

Normal Learning: The teacher gives every mistake the same "penalty." If the student misses a dog, they get a small red mark. If they miss a tiger, they get the same small red mark. Since there are so many dogs, the student focuses on getting the dogs right to get a good grade, ignoring the tigers.
Weighted Loss: The teacher turns up the volume on the rare mistakes. Now, if the student misses a tiger, the penalty is huge (a big red "X"). If they miss a dog, the penalty is tiny. This forces the student to pay attention to the rare tigers so they don't get a terrible score.

The authors applied this "volume knob" to three different types of AI brains (called BERT, RoBERTa, and BART) to see if it helped them hear the quiet emotions better.

The Results: Who Listened?

The experiment had some surprising twists:

BERT and BART (The Eager Students): These models were very sensitive to the imbalance. Without the "volume knob," they struggled with rare emotions. When the authors turned up the volume (added the weighted loss), these models got significantly better at spotting all emotions, including the rare ones. It was like giving them a pair of hearing aids; their overall performance jumped up.
RoBERTa (The Already-Expert Student): This model was already so well-trained that it didn't need the "volume knob" as much. In fact, when the authors turned up the volume, RoBERTa's performance barely changed, and in some specific measurements, it actually got slightly worse. It seems this model was already naturally good at balancing the loud and quiet emotions on its own.

The Catch: The "High-Frequency" Boost

Here is the most important finding: While the "volume knob" helped the models get better overall, it didn't magically solve the problem for the rarest emotions.

The authors found that the improvement mostly came from the models getting even better at spotting the common, frequent emotions (like "joy"). The rare emotions (like "fear" or "surprise") didn't see much of a boost. It's as if the volume knob made the student pay more attention to the dogs, but they still weren't quite sure what a tiger looked like.

The Conclusion

The paper concludes that using a "weighted loss" (turning up the volume on rare mistakes) is a simple and effective tool. It helps models like BERT and BART perform much better overall. However, it is not a magic wand that completely fixes the difficulty of spotting rare emotions. The AI still struggles a bit with the "quiet" feelings, suggesting that while this method helps, the challenge of teaching machines to understand subtle, rare human emotions is still ongoing.

In short: The authors tried to make AI pay more attention to rare emotions by giving them "extra credit" for getting them right. It worked well for some AI models, making them smarter overall, but it didn't completely solve the mystery of the rarest emotions.

Technical Summary: Addressing Data Imbalance in Transformer-Based Multi-Label Emotion Detection with Weighted Loss

Problem Definition
The paper addresses the challenge of multi-label emotion detection in short texts, a task complicated by the inherent ambiguity of emotional language and the frequent co-occurrence of multiple emotions within a single instance. A critical barrier to effective automated detection is data imbalance, where certain emotion classes (majority classes) significantly outnumber others (minority classes). Traditional resampling techniques (oversampling or undersampling) are often ineffective in multi-label scenarios because altering the frequency of one label can inadvertently disrupt the complex mapping of instances to multiple simultaneous output labels. The authors frame this as a multi-label classification problem where the goal is to predict a subset of emotion labels (anger, fear, joy, sadness, surprise) for a given text input.

Methodology
The proposed approach utilizes Transformer-based architectures (specifically BERT, RoBERTa, and BART) as the backbone for feature extraction and classification. To address data imbalance without the computational overhead of resampling, the authors introduce a simple weighted loss function.

Architecture: The models employ self-attention mechanisms and positional encodings to capture contextual relationships. The output layer consists of independent binary classifiers for each emotion class, utilizing a sigmoid activation function.
Loss Function Modification: The standard Binary Cross-Entropy (BCE) loss is modified by incorporating class weights ( $w_j$ ). These weights are dynamically adjusted based on the frequency of each class in the training dataset ( $w_j = \text{max}(W) / w_j$ , where $W$ represents the distribution of labels). This mechanism increases the penalty for misclassifying minority classes during training.
Regularization: The method also employs label smoothing to prevent model overconfidence and reduce overfitting on majority classes.
Experimental Setup: Experiments were conducted on the BRIGHTER dataset (English subset), which contains multi-label annotations across five emotion classes. The dataset was split into training (70%) and development (30%) sets. The authors utilized Optuna for Bayesian hyperparameter optimization to maximize the Macro F1 score, a metric robust to class imbalance. All models were trained using the AdamW optimizer with early stopping.

Key Contributions
The paper outlines three primary contributions:

Proposal of a Weighted Loss Strategy: The authors propose a straightforward weighted loss function specifically designed to handle data imbalance in multi-label emotion detection using Transformer models, avoiding the complexities of resampling.
Comprehensive Model Evaluation: The study evaluates three prominent Transformer models—BERT, RoBERTa, and BART—to determine their efficacy in this specific multi-label context.
Empirical Analysis of Weighting Impact: The work provides a detailed analysis of how the weighted loss function (+w) affects different models and emotion classes, revealing nuanced performance shifts rather than uniform improvements.

Results
The evaluation utilized Micro F1, Macro F1, ROC-AUC, Accuracy, and Jaccard similarity coefficients. The results indicate that the impact of the weighted loss function is model-dependent:

BERT and BART: Both models showed notable improvements with the weighted loss function. Specifically, BART+w achieved the most significant gains, with Micro F1 increasing from 0.6961 to 0.7321 and Weighted F1 rising from 0.6993 to 0.7350. BERT also saw consistent improvements across all metrics.
RoBERTa: In contrast, RoBERTa showed limited benefit and slight underperformance in Micro F1 (dropping from 0.7282 to 0.7268) when the weighted loss was applied. The authors suggest RoBERTa is inherently well-optimized for class imbalance due to its robust pretraining, making it less sensitive to the reweighting strategy.
Class-Specific Performance: Further analysis (Tables 2 and 3) revealed that improvements in Macro F1 and Sample F1 were primarily driven by gains in high-frequency classes (e.g., joy, surprise). The weighted loss function showed limited or no improvement for most minority emotion classes (4 out of 5 classes showed stagnation or decline in specific metrics).
Confusion Patterns: The study observed overlapping misclassifications between "joy" and "surprise," attributed to contextual ambiguity, highlighting a limitation in current text-based representations for fine-grained emotional distinctions.

Significance and Claims
The paper modestly claims that while the weighted loss function is an effective strategy for improving performance on high-frequency emotion classes and enhancing overall robustness for certain architectures (BERT and BART), it does not fully resolve the challenges of detecting minority classes. The findings underscore that while the approach mitigates some biases introduced by imbalanced data, the detection of rare emotions remains a significant hurdle. The authors conclude that future work should explore the generalizability of this approach across different languages and datasets, rather than claiming a definitive solution to the multi-label imbalance problem. The source code is made publicly available to facilitate further research.