💬 NLP

MEME-Fusion@CHiPSAL 2026: Multimodal Ablation Study of Hate Detection and Sentiment Analysis on Nepali Memes

This paper introduces MEME-Fusion, a hybrid cross-modal attention architecture that significantly outperforms text-only baselines in detecting hate speech and sentiment in low-resource Nepali memes by dynamically fusing CLIP and BGE-M3 encodings, while revealing critical limitations of English-centric vision models and standard ensembles in Devanagari-scripted, data-scarce environments.

Original authors: Samir Wagle, Reewaj Khanal, Abiral Adhikari

Published 2026-04-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Samir Wagle, Reewaj Khanal, Abiral Adhikari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a moderator for a massive, chaotic digital town square. In this square, people aren't just shouting words; they are posting memes. These memes are like inside jokes wrapped in a picture, where the text and the image work together to convey a message. Sometimes, that message is funny; other times, it's hate speech.

Your job is to spot the hate. But there's a catch: the town square is speaking Nepali (using the Devanagari script), and you only have a tiny, tattered notebook of examples (about 850 posts) to learn from. Most of the AI tools you have were trained on English and Western pictures, so they are confused by the local script and culture.

This paper is the story of how a team of researchers built a new "detective" to solve this specific problem. Here is the breakdown of their journey:

1. The Problem: The "Blind" Detective

The researchers tried using standard AI tools, but they hit two major walls:

The Language Barrier: The famous AI "eyes" (called CLIP) are like a tourist who only knows English. When they look at a Nepali meme, they can't read the text on the sign. They just see shapes and colors, guessing wildly. In fact, when the team let the AI look only at the pictures, it performed no better than a monkey throwing darts at a dartboard.
The Data Scarcity: Usually, AI learns by reading millions of books. Here, the team only had a few hundred pages. If you try to teach a complex AI with too little data, it gets confused and starts "cheating" (overfitting), memorizing the few examples instead of learning the rules.

2. The Solution: The "Dynamic Duo" Detective

Instead of using one big, confused brain, they built a Hybrid Detective with two partners who talk to each other:

Partner A (The Reader): An expert in Nepali text who can understand the words, slang, and emojis.
Partner B (The Observer): An expert in images who looks at the faces, colors, and layout.

The Secret Sauce: The "Gating Network"
Most AI systems just mash the text and image together into a big pile of data. This team did something smarter. They added a Traffic Cop (a "gating network") between the two partners.

If the text is clearly hate speech (like a direct slur), the Traffic Cop says, "Ignore the picture, the words are the smoking gun!"
If the text is vague but the picture shows a hateful symbol, the Traffic Cop says, "The words are quiet, but the image is screaming! Focus on the picture!"

This allows the system to adapt to every single meme individually, rather than using a "one-size-fits-all" approach.

3. The Surprising Discoveries

The team ran experiments to see what worked and what failed. They found some counter-intuitive truths:

Ensembles Don't Always Help: Usually, if you ask three experts for their opinion and take the average, you get a better answer. But with so little data, the three experts all made the same mistakes. Instead of canceling out errors, they amplified them. It was like asking three people who all forgot their glasses to describe a blurry sign; they just agreed on the wrong description.
Accuracy is a Trap: The "Text-Only" detective was actually more accurate at guessing right overall (72% accuracy) than the "Hybrid" detective. But, the Text-Only detective cheated by just guessing "Hate" for everything, because most memes in their small dataset were labeled as hate. It missed the innocent memes entirely. The Hybrid detective was slightly less "accurate" overall but was much fairer, catching the hate and correctly identifying the innocent posts. In the real world, fairness matters more than raw guessing speed.

4. The Results

The team entered their Hybrid Detective into a global competition (CHiPSAL 2026).

For finding Hate Speech: They came in 4th place.
For analyzing Sentiment (Happy/Neutral/Sad): They came in 5th place.

More importantly, their system proved that when you have very little data and a complex language, you can't just throw big, expensive AI models at the problem. You need a smart, flexible system that knows when to listen to the words and when to look at the picture.

The Takeaway

This paper teaches us that in the world of low-resource languages (like Nepali), context is king. You can't just translate an English AI and hope it works. You need a system that understands the local culture, knows when to trust the text over the image (and vice versa), and doesn't get tricked by small datasets.

It's like teaching a child to recognize a "stop sign." You don't just show them a picture of a red octagon; you have to explain that in this specific neighborhood, the sign might be drawn on a wall, or the text might be in a different language, but the meaning remains the same. That is what this "Hybrid Fusion" system does.

1. Problem Statement

The paper addresses the critical challenge of detecting hate speech and analyzing sentiment in Nepali memes (social media images containing Devanagari script). The problem is characterized by three specific structural difficulties:

Extreme Data Scarcity: The dataset contains only 1,068 total instances (approx. 850 per training fold), creating a severe sample-to-parameter ratio that destabilizes high-dimensional models.
Class Imbalance: Subtask A (Hate Detection) has a 2.07:1 ratio of Hate to Non-Hate, while Subtask B (Sentiment) is skewed toward Negative and Neutral classes.
Linguistic and Cultural Gap: Standard Vision-Language Models (VLMs) like CLIP are pre-trained on English-centric data, rendering them ineffective for Devanagari script and culturally specific Nepali visual contexts.

The authors participated in the CHiPSAL 2026 shared task, tackling two subtasks:

Subtask A: Binary Hate Speech Detection (Hate vs. Non-Hate).
Subtask B: Three-class Sentiment Classification (Positive, Neutral, Negative).

2. Methodology

The authors propose a Hybrid Cross-Modal Attention Fusion Architecture designed to dynamically weigh text and visual inputs on a per-sample basis.

A. Encoders

Visual Encoder: CLIP (ViT-B/32). To mitigate overfitting on the small dataset, the authors froze the first 10 layers and fine-tuned only the final 2 transformer layers.
Text Encoder: BGE-M3. Selected over E5-large and XLM-RoBERTa for its superior performance on Indic scripts and low-resource retrieval. The final 4 layers were fine-tuned to adapt semantic representations to the hate speech domain.

B. Fusion Strategies (Ablation Study)

The study systematically evaluated eight model configurations across three paradigms:

Unimodal Baselines: Text-only (M1) and Image-only (M2/M3).
Conventional Fusion:
- Early Fusion (M4): Direct concatenation of features.
- Late Fusion (M5/M6): Soft voting and Bagging ensembles.
Proposed Hybrid Architecture (M7/M8):
- Dimensionality Reduction: Text and visual embeddings are projected into a shared, lower-dimensional latent space to prevent text dominance.
- Cross-Modal Attention: A 4-head self-attention layer computes explicit dependencies between modalities.
- Learnable Gating Network: A softmax-normalized gate dynamically assigns weights to text vs. visual features for each specific sample. This allows the model to prioritize text when linguistic slurs are explicit and upweight visual features when text is ambiguous.

C. Training & Preprocessing

Loss Function: Frequency-weighted cross-entropy with a suppression exponent (0.3) to handle class imbalance.
Regularization: Heavy use of dropout (0.5), label smoothing (0.1), and early stopping to prevent overfitting on the small dataset.
Data Augmentation: A "Text-Removed" image variant was created using Gaussian blur inpainting to isolate the contribution of visual layout from typography.

3. Key Contributions & Findings

The paper provides significant insights beyond the primary system performance:

Superiority of Dynamic Fusion: The proposed Hybrid Fusion model (M7) outperformed all baselines. It achieved a 5.9% absolute improvement in Macro-F1 over the text-only baseline on Subtask A.
Failure of Standard Ensembles in Low-Resource Settings: Counter-intuitively, standard ensemble methods (Bagging and Soft Voting) degraded performance compared to simpler models. Due to the small dataset size ( $N \approx 850$ ), bootstrap resampling created highly correlated folds, causing the ensemble to amplify noise rather than cancel it out.
CLIP's Limitation on Devanagari: Image-only models achieved near-random performance (Macro-F1 $\approx$ 0.55). The study found that CLIP relies on weak visual heuristics (e.g., text density) rather than understanding Indic semantics, highlighting a critical gap in current VLM pretraining for South Asian scripts.
The "Accuracy-F1 Paradox": The text-only baseline achieved higher accuracy (0.7239) but lower Macro-F1 (0.6455) because it aggressively over-predicted the majority "Hate" class. The proposed model (M7) sacrificed raw accuracy (0.7015) to significantly improve Non-Hate recall (from 0.59 to 0.73), preventing over-censorship of benign content.

4. Results

The system was evaluated on the CHIPSAL 2026 leaderboard via 5-fold cross-validation.

Metric	Subtask A (Hate Detection)	Subtask B (Sentiment)
Best Model	M7 (Hybrid Fusion)	M7 (Hybrid Fusion)
Macro-F1	0.6834	0.6102
Ranking	4th Place	5th Place
Improvement vs. Text-Only	+5.9% F1	+2.2% F1

Subtask A: M7 improved Non-Hate recall by 14 percentage points compared to the text-only baseline, demonstrating better balance in detecting minority classes.
Subtask B: The task was harder due to the three-class nature (Positive/Neutral/Negative), but the hybrid architecture still maintained the performance hierarchy (Hybrid > Late Voting > Text-Only).

5. Significance and Implications

Architectural Design for Low-Resource NLP: The paper demonstrates that in data-scarce regimes, constrained, dynamic fusion (M7) is mathematically superior to naive concatenation or complex ensembles, which suffer from the curse of dimensionality and correlated overfitting.
Cultural Adaptation: It exposes the inadequacy of English-centric VLMs for Devanagari content, suggesting that future South Asian content moderation requires natively multilingual vision models (e.g., IndicCLIP).
Evaluation Metrics: The study argues that Macro-F1 and class-specific recall are more critical than overall accuracy for hate speech detection, as accuracy can mask systemic failures in identifying non-hateful content (over-censorship).
Error Analysis: The system struggles with political satire (false positives) and cultural masking (false negatives where hate is hidden via code-mixed OOV tokens), indicating a need for models that understand sociolinguistic context, not just semantics.

Conclusion

The MEME-Fusion system successfully addresses the challenges of multimodal hate detection in low-resource, non-Latin script environments by combining a robust text encoder (BGE-M3) with a carefully regularized visual encoder (CLIP) via a dynamic gating mechanism. The work serves as a cautionary tale against applying standard ensemble methods to small datasets and a blueprint for building culturally aware, multimodal moderation systems.