💻 computer science

Cost-sensitive multi-label classification in BERT-based models: An inverse-frequency re-weighting approach to emoji pragmatics

This study demonstrates that applying an inverse-frequency re-weighting approach to MARBERT significantly outperforms baseline models in classifying underrepresented emoji pragmatic functions within Arabic digital discourse, effectively addressing severe label imbalance to advance computational pragmatics in Arabic NLP.

Original authors: Fahad Al Hussen, Mohammed Shormani

Published 2026-09-08

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Fahad Al Hussen, Mohammed Shormani

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, noisy landscape of social media, a single post can carry a dozen different meanings at once. A sentence might be a joke, a sigh of relief, and a subtle dig at a friend, all wrapped up in a few words and a handful of colorful icons. For computers, understanding this complexity is a monumental task. While machines have become quite good at spotting simple emotions like happiness or anger, they often stumble when asked to recognize the more nuanced, layered purposes of human communication. This is the realm of digital pragmatics: the study of how people use language and symbols to do things, like show solidarity, soften a blow, or express sarcasm. In the Arabic-speaking world, this challenge is even greater because the language itself is a living mosaic. People switch effortlessly between a formal, written standard and a variety of colorful, informal dialects, often mixing them within a single conversation. When you add emojis to this mix, the task of teaching a computer to understand the full intent behind a post becomes a puzzle of immense difficulty.

Two researchers, Fahad Al Hussen and Mohammed Shormani, set out to solve a specific piece of this puzzle. They wanted to know if advanced computer models could learn to identify the multiple, overlapping functions of emojis in Arabic Facebook posts, even when some of those functions appeared very rarely. They focused on a dataset of over 8,000 unique posts, each containing at least two emojis. The goal was not just to say whether a post was positive or negative, but to classify it into specific categories of meaning, such as "prayer," "surprise," "agreement," or "mitigation" (the act of softening a disagreement). The researchers faced a significant hurdle: in their data, some functions like "humor" appeared thousands of times, while others like "mitigation" appeared only a few dozen times. This imbalance is a common problem in artificial intelligence; without help, a computer tends to ignore the rare items and focus only on the common ones, effectively becoming blind to the subtle, less frequent meanings.

To tackle this, the team tested two powerful computer models designed to understand Arabic text. The first, known as AraBERT, was trained primarily on formal, standard Arabic, much like a student who has studied textbooks and news articles. The second, MARBERT, was trained on a massive collection of social media posts, making it more familiar with the slang, dialects, and informal writing styles found in everyday online life. The researchers taught these models to recognize ten different micro-functions of emojis, ranging from expressing love to signaling sarcasm. However, they knew that simply feeding the data to the models would not be enough. To fix the problem of the rare functions being ignored, they introduced a clever adjustment to the learning process. Instead of treating every example of a function as equally important, they made the computer pay extra attention to the rare ones. Imagine a teacher grading a student's homework: if the student gets the common questions right but misses the rare, difficult ones, the teacher might decide to give those rare questions extra weight in the final score to ensure the student truly understands the whole subject. The researchers applied this same logic, telling the computer that making a mistake on a rare function like "mitigation" was a much bigger error than making a mistake on a common one like "emphasis."

The results of this experiment were revealing. When the models were left to learn without this special adjustment, they performed reasonably well on the common functions but failed almost completely on the rare ones. For instance, the models could not identify "surprise," "agreement," or "mitigation" at all, assigning them a success rate of zero. This confirmed that without intervention, the computer's focus on the frequent data blinded it to the subtle, infrequent meanings. However, once the researchers applied their cost-sensitive re-weighting method, the picture changed dramatically. The models began to recognize the rare functions with much greater accuracy. The model trained on social media data, MARBERT, showed the most impressive improvement. Its ability to correctly identify the full range of functions jumped significantly, moving from a moderate level of success to a high level of balanced performance. It learned to spot "surprise" and "agreement" with much greater reliability, and it even began to catch the elusive "mitigation" function, which had previously been invisible to the system.

The study also highlighted a crucial difference between the two models. The model trained on formal Arabic, AraBERT, did improve with the new method, but its gains were modest. It struggled to adapt to the messy, mixed nature of the social media posts. In contrast, MARBERT, which had been exposed to the real-world chaos of dialects and informal speech during its training, was far better equipped to handle the task. This suggests that for understanding how people actually communicate online, a model needs to be trained on the language as it is spoken and typed in those spaces, not just how it appears in formal books. The researchers found that combining a model trained on social media with a learning strategy that forces attention to rare examples produced the best results. They did not find that one method alone was a magic solution; rather, the success came from the combination of a model that understood the linguistic context and a training method that ensured no part of the conversation was ignored.

Ultimately, this work demonstrates that computers can be taught to understand the complex, multi-layered ways people use emojis in Arabic digital discourse, provided the training process is designed to value the rare and the common equally. The researchers showed that by adjusting how the computer learns, they could unlock the ability to recognize subtle pragmatic functions that were previously missed. This is a significant step forward for Arabic language technology, moving beyond simple sentiment analysis to a deeper understanding of human interaction. The findings suggest that for artificial intelligence to truly grasp the richness of digital communication, it must be trained on the diverse, informal, and often unbalanced reality of how people actually speak and write online. The study concludes that while the technology is not perfect and still faces challenges with extremely rare examples, the path forward lies in using models that reflect the true diversity of the language and in designing learning systems that do not let the common overshadow the unique.

Problem Statement

The study addresses a significant gap in Arabic Natural Language Processing (NLP) regarding the computational modeling of digital pragmatics. While existing research in Arabic NLP has largely focused on single-label sentiment analysis, emotion detection, or sarcasm, it often fails to capture the communicative complexity of Arabic Digital Discourse (ADD). In ADD, particularly on platforms like Facebook, a single post can simultaneously realize multiple, non-mutually-exclusive pragmatic functions through the interaction of text and emojis.

The core challenges identified are:

Multi-Label Complexity: Emoji usage often embodies both macro-pragmatic functions (Affective and Interactional) and fine-grained micro-functions (e.g., prayer, surprise, mitigation) that co-occur within the same context.
Severe Class Imbalance: Pragmatic functions in ADD are highly imbalanced. Frequent functions (e.g., emphasis, humor) dominate the dataset, while rare but theoretically significant functions (e.g., mitigation, agreement, surprise) are underrepresented. Standard classification models tend to ignore these minority classes, leading to poor balanced performance.
Linguistic Diversity: Arabic is a diglossic language, and ADD is characterized by a mix of Modern Standard Arabic (MSA) and various Dialectal Arabic (DA) varieties, informal orthography, and noise, which poses challenges for pre-trained models trained primarily on formal corpora.

Methodology

Data Collection and Annotation

Corpus: The authors collected 8,390 unique Arabic Facebook posts containing two or more emojis. Posts with only one emoji or no meaningful text were excluded to ensure sufficient contextual ambiguity for pragmatic interpretation.
Annotation Scheme: The corpus was manually annotated by three trained annotators using a hierarchical framework. Each post was assigned a 10-dimensional binary vector representing 10 micro-functions:
- Affective Domain: Pray, Sadness, Surprise, Anger, Love.
- Interactional Domain: Agreement, Mitigation, Emphasis, Humor, Sarcasm.
Split: The data was partitioned into training (70%), validation (15%), and evaluation (15%) sets.

Models
Two Arabic BERT-based Transformer models were employed and compared:

AraBERT: Pre-trained on formal, Modern Standard Arabic (MSA) corpora (news, Wikipedia).
MARBERT: Pre-trained on large-scale, dialectal Arabic (DA) social media data, designed to handle informal and noisy user-generated content.

Proposed Approach: Inverse-Frequency Re-weighting (IFRW)
To address label imbalance without altering the original data distribution (i.e., avoiding resampling), the authors introduced a Cost-Sensitive Multi-Label Classification (CSMLC) approach using Positive-Class Inverse-Frequency Re-weighting.

Mechanism: A weight ( $W_c$ ) is calculated for each micro-function $c$ based on the ratio of negative to positive instances in the training set: $W_c = N_{negative,c} / N_{positive,c}$ .
Implementation: These weights are integrated into the Binary Cross-Entropy (BCE) loss function during the fine-tuning process. Rare functions (e.g., Mitigation with only 40 positive instances) receive high weights (e.g., 145.80), while frequent functions (e.g., Emphasis) receive weights below 1.0.
Algorithms: The study introduces two algorithms:
1. Algorithm 1: Computes the IFRW weight vector from the training distribution.
2. Algorithm 2: The cost-sensitive fine-tuning procedure that incorporates these weights into the loss function and optimizes the model using the AdamW optimizer.

Experimental Setup
Both models were fine-tuned for 5 epochs with a learning rate of $2 \times 10^{-5}$ and a batch size of 16. Evaluation metrics included Macro- and Micro-averaged Precision, Recall, F1-score, Jaccard Accuracy, and Hamming Loss.

Key Results

Overall Performance

Baseline vs. Reweighted: The IFRW approach significantly improved balanced performance. MARBERT-ReW achieved the highest overall metrics: Macro-F1 of 0.75, Micro-F1 of 0.88, and Jaccard Accuracy of 0.87.
Model Comparison: MARBERT consistently outperformed AraBERT. Under reweighting, MARBERT's Macro-F1 increased from 0.53 to 0.75, whereas AraBERT's increased modestly from 0.50 to 0.54.
Metric Discrepancy: Baseline models showed high Micro-F1 (dominated by frequent classes) but low Macro-F1, indicating they failed to recognize rare functions. Reweighting narrowed this gap, demonstrating more equitable recognition across all functions.

Function-Level Performance
The most significant gains were observed in rare functions that baseline models failed to detect (F1 = 0.00):

Surprise: Improved from 0.00 to 0.73 (MARBERT-ReW).
Agreement: Improved from 0.00 to 0.56.
Anger: Improved from 0.00 to 0.37.
Mitigation: Improved from 0.00 to 0.19 (remained the most difficult due to extreme scarcity of only 40 training instances).
Trade-off: There was a minor decrease in performance for frequent functions (e.g., Emphasis and Humor dropped slightly from ~0.90 to ~0.89), reflecting the optimization trade-off inherent in cost-sensitive learning.

Key Contributions

Extension of Arabic NLP Scope: The study moves beyond sentiment and emotion analysis to model emoji pragmatic functions as a multi-label classification problem, capturing both affective and interactional dimensions in ADD.
Cost-Sensitive Optimization for Pragmatics: It introduces and validates an Inverse-Frequency Re-weighting (IFRW) strategy specifically for severe class imbalance in multi-label pragmatic tasks, preserving the original data distribution while increasing learning pressure on minority classes.
Empirical Model Comparison: It provides a controlled comparison between AraBERT (MSA-oriented) and MARBERT (DA-oriented), offering evidence that dialect-aware pre-training is superior for social media pragmatic analysis.
Evaluation Framework: It highlights the necessity of using Macro-averaged metrics alongside Micro-averaged ones to reveal model deficiencies in recognizing rare but theoretically important pragmatic functions.

Significance and Claims

The authors claim that their study bridges the gap between deep learning and digital pragmatics in Arabic by demonstrating that:

Transformer-based models can learn complex, co-occurring pragmatic functions when paired with cost-sensitive optimization.
MARBERT is a more robust backbone for socio-pragmatic analysis in Arabic digital discourse due to its exposure to dialectal and informal language, outperforming MSA-focused models like AraBERT in this specific context.
IFRW is an effective mechanism for recovering minority classes without the data distortion caused by resampling techniques.

The study concludes that effective multi-label pragmatic classification requires a combination of high-quality contextual representations (provided by MARBERT) and imbalance-aware optimization (provided by IFRW).

Limitations

The authors modestly acknowledge several limitations:

Data Scarcity: Extremely rare classes (like Mitigation) still suffer from limited positive instances, suggesting that reweighting alone cannot fully substitute for adequate corpus representation.
Tokenization Challenges: The diglossic nature of Arabic and informal orthography (e.g., character elongation) introduce noise that standard subword tokenizers handle imperfectly.
Flat Architecture: The study uses a flat multi-label framework rather than hierarchical classifiers, meaning it does not explicitly model the conditional dependencies between macro and micro functions.
Scope: The taxonomy is limited to affective and interactional functions, excluding broader socio-pragmatic dimensions like politeness strategies or power dynamics.