💬 NLP

ZeroR@CHiPSAL 2026: Two-Stage Vision-Language Adaptation with Contrastive Learning for Nepali Meme Classification

The ZeroR@CHiPSAL 2026 team achieved top rankings in Nepali meme classification by employing a two-stage vision-language adaptation pipeline using Qwen3-VL-8B-Instruct with contrastive learning and LoRA fine-tuning to eliminate OCR dependencies and effectively handle low-resource multimodal hate speech and sentiment detection.

Original authors: Nitiz Khanal

Published 2026-08-03

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitiz Khanal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, chaotic digital town square where everyone is shouting, whispering, and drawing pictures on the walls. In this town, "memes" are the most popular way to communicate. They are like inside jokes that mix a funny picture with a clever caption. But sometimes, these jokes turn into weapons, hiding hate speech or bullying inside a laugh. The tricky part is that a picture might look innocent, but the text makes it mean something terrible, or vice versa. To stop this, computers need to learn how to "read" both the image and the words at the same time, understanding the whole story, not just the parts. This is a huge challenge for computers, especially when the jokes are told in languages they don't know well, like Nepali, where the writing system (called Devanagari) is drawn directly onto the pictures.

Scientists have been trying to build better "digital detectives" to spot these harmful memes. Usually, they try to build a long assembly line: first, a machine reads the text from the picture (like a scanner), then another machine translates it, and finally, a third machine decides if it's bad. But this is like trying to solve a puzzle by taking it apart piece by piece; you might lose the picture of the whole thing. This paper introduces a new team of detectives that doesn't need the assembly line. Instead, they use a super-smart, all-in-one brain that can look at the picture and read the text simultaneously, just like a human does. The researchers tested this brain on Nepali memes to see if it could spot hate speech and figure out if the mood was happy, sad, or neutral.

The Paper's Story: A Two-Stage Detective Training

The researchers, Nitiz Khanal and his team, entered a competition called CHiPSAL 2026, where the goal was to build the best system for spotting hate and analyzing feelings in Nepali memes. They didn't just throw a computer at the problem; they designed a clever two-stage training camp for their AI model.

The Star Player: Qwen3-VL
At the heart of their system is a massive, pre-trained AI called Qwen3-VL-8B-Instruct. Think of this model as a genius student who has already read millions of books and seen millions of pictures from around the world. Crucially, this student already knows how to read the Devanagari script used in Nepal, so the researchers didn't need to teach it how to read from scratch or use a separate tool to scan the text. It could look at a meme and "see" the words written on the image immediately.

Stage 1: The Creative Writer
In the first stage of training, the team taught the AI to act like a creative writer. They showed it thousands of memes and asked it to write down the answer: "Hate" or "Safe," or "Negative," "Neutral," or "Positive." To do this without breaking the computer, they used a technique called LoRA (Low-Rank Adaptation). Imagine the AI is a giant library with millions of books. Instead of rewriting every single book to teach it about Nepali memes, the researchers just added a few sticky notes (small, trainable layers) to the shelves. This allowed the AI to learn the new task quickly and efficiently.

However, the data they had was messy. There were way more "Safe" memes than "Hate" ones, and way more "Neutral" memes than "Positive" or "Negative" ones. It's like trying to learn to identify rare animals when you only have one photo of a tiger but a thousand photos of cats. To fix this, the team used oversampling (making extra copies of the rare memes) and image augmentation (flipping, rotating, and brightening the images) so the AI wouldn't just memorize the pictures but actually learn the patterns. For the three-class sentiment task, they also used a special math trick called Focal Loss, which told the AI to pay extra attention to the difficult examples it kept getting wrong.

Stage 2: The Strict Coach
The first stage was good, but the researchers wanted the AI to be even sharper. In the second stage, they added a "contrastive learning" coach. This is like a game of "spot the difference." The coach showed the AI two memes that were both "Hate" and said, "These two belong together!" Then it showed a "Hate" meme and a "Safe" meme and said, "These two are totally different! Push them apart!" This helped the AI learn to group similar ideas tightly together and push different ideas far apart in its mind. This stage didn't just ask for an answer; it forced the AI to understand the shape of the data, making its decisions more robust.

The Final Verdict: Mixing the Scores
When it was time for the final exam, the team didn't just pick the answer from Stage 1 or Stage 2. They created a team huddle. They took the probability score from the "Creative Writer" (Stage 1) and the score from the "Strict Coach" (Stage 2) and mixed them together with a special weight. They tested different mixing ratios on a practice set to find the perfect balance that gave the highest score.

What They Found

The results were impressive. The team's system achieved 2nd place on the official leaderboard for the hate speech detection task, with a score (called Macro F1) of 0.797. This means the system was very good at spotting the harmful content, even with the tricky class imbalance. For the sentiment analysis task (figuring out if a meme was positive, negative, or neutral), they placed 4th on the leaderboard with a score of 0.518.

The researchers found that the two-stage approach was the secret sauce. If they had only used the first stage (just writing the answer), their score would have been lower. Adding the second stage (the contrastive learning) boosted their performance significantly. They also discovered that the two tasks needed different training styles. The hate speech task was a bit more straightforward, but the sentiment task was much harder to learn because the lines between "neutral" and "positive" or "negative" are often blurry, especially with humor. The sentiment model needed much stricter rules (regularization) to stop it from getting confused.

Why This Matters (and What It Doesn't)

This paper suggests that for low-resource languages like Nepali, using a powerful, all-in-one vision-language model that natively understands the script is a game-changer. It proves that you don't need a complicated chain of tools (like separate text scanners and translators) to understand memes; a single, smart model can do it end-to-end.

However, the author is careful to point out that their system isn't perfect. They admit that the AI sometimes misses the deep cultural context. For example, a meme might feature a historical figure or a specific cultural joke that looks harmless to the AI but is actually offensive to a Nepali person. The AI is smart, but it doesn't have the lived experience of the culture yet. They also note that because the dataset was relatively small (around 1,000 memes per task), there's a risk the model might have memorized the training data rather than truly learning, and the results might vary if they started over with a different random seed.

In the end, this work shows a promising path forward: teaching big, smart AI models to understand the messy, funny, and sometimes dangerous world of memes in languages that have been left behind by technology. It's a step toward making the digital town square safer and more inclusive for everyone.

Technical Summary: ZeroR@CHiPSAL 2026

Problem Statement

The paper addresses the challenge of multimodal hate speech and sentiment detection in Nepali memes, a low-resource language context characterized by text embedded directly within images using the Devanagari script. Unlike traditional text or image classification, meme analysis requires understanding the complex interplay between visual and textual elements, where meaning often emerges from their combination. The specific challenges include:

Low-Resource Constraints: Scarcity of annotated datasets and limited pre-trained model exposure to Nepali cultural context.
Script Complexity: Nepali text is embedded in images, requiring Optical Character Recognition (OCR) in traditional pipelines, which introduces error propagation.
Code-Mixing: Frequent mixing of Nepali and English in social media content.
Task Specificity: The shared task involves two distinct subtasks: binary hate speech detection (Hate vs. Safe) and three-class sentiment analysis (Negative, Neutral, Positive), both suffering from significant class imbalance.

Methodology

The author proposes a two-stage vision-language adaptation framework based on the Robust Adaptation of Hateful Meme Detection (RA-HMD) architecture, utilizing Qwen3-VL-8B-Instruct as the backbone. This model was selected for its native support of the Devanagari script, enabling end-to-end processing without separate OCR or translation pipelines.

1. Base Model and Adaptation

Backbone: Qwen3-VL-8B-Instruct, capable of dynamic image resolution and instruction following.
Parameter-Efficient Fine-Tuning (PEFT): The system employs Low-Rank Adaptation (LoRA) to inject trainable matrices into the transformer layers, freezing the pre-trained weights. This reduces trainable parameters to approximately 0.5% for hate speech and 0.25% for sentiment analysis.
MLP Projection: An MLP head maps the model's last hidden state to a lower-dimensional embedding space (512 dimensions) to facilitate contrastive learning.

2. Two-Stage Training Pipeline

The training process is divided into two distinct stages:

Stage 1: Generative Fine-Tuning
- The model is fine-tuned to generate class labels directly from meme images using task-specific prompts.
- Objective: Standard language modeling loss computed only over target tokens (e.g., "Hate" or "Safe").
- Imbalance Handling:
  - Oversampling: Minority classes are duplicated to achieve balanced ratios (1:1 for hate speech; 1:1:1 for sentiment).
  - Augmentation: Oversampled images undergo random transformations (flips, rotation, brightness/contrast/saturation adjustments).
  - Focal Loss: Applied specifically to the three-class sentiment task to reduce the loss contribution from easy examples and focus on hard cases.
Stage 2: Contrastive Backbone Fine-Tuning
- Training continues with a joint objective combining classification and supervised contrastive learning.
- Objective: A linear classifier is added to the projected embeddings, and the Supervised InfoNCE loss is applied. This pulls together representations of samples from the same class while pushing apart different classes.
- Configuration: The sentiment task required stronger regularization (lower learning rate, higher weight decay, fewer epochs) compared to the hate speech task to prevent overfitting on the smaller effective training set per class.

3. Inference and Ensemble

At inference, the system ensembles predictions from both stages:

Stage 1: Extracts token-level probabilities from the language model head.
Stage 2: Applies softmax to the linear classifier outputs.
Ensemble: The final prediction is a weighted combination ( $w \cdot p^{(1)} + (1-w) \cdot p^{(2)}$ ). The weight $w$ and decision thresholds are tuned via grid search on the validation set to optimize the macro F1-score.

Key Contributions

Two-Stage Framework: Adaptation of the RA-HMD framework for low-resource multimodal classification, combining generative fine-tuning with contrastive learning.
End-to-End Processing: Leveraging native Devanagari support in Qwen3-VL to eliminate error-prone OCR and translation pipelines.
Task-Specific Configurations: Distinct hyperparameter settings and regularization strategies tailored to the binary nature of hate speech detection versus the complexity of three-class sentiment analysis.
Imbalance Mitigation: A comprehensive strategy involving minority oversampling, image augmentation, and focal loss to address severe class imbalances.

Results

The system was evaluated on the CHiPSAL 2026 shared task using the macro F1-score:

Subtask A (Hate Speech Detection): Achieved an F1-score of 0.797, securing 2nd place.
Subtask B (Sentiment Analysis): Achieved an F1-score of 0.518, securing 4th place.

Ablation Findings:

Applying LoRA to all attention and feed-forward layers outperformed applying it only to attention layers.
Class imbalance handling (oversampling + augmentation) contributed 2–3 F1 points.
The addition of Stage 2 contrastive learning provided consistent gains (approx. 2 F1 points for hate speech).
Threshold tuning improved hate speech detection by an additional 1 point.

Significance and Claims

The paper claims that its primary significance lies in demonstrating the effectiveness of adapting modern, large Vision-Language Models (VLMs) with native script support for low-resource South Asian languages. Key takeaways include:

Elimination of Error Propagation: By using a model with native Devanagari understanding, the system avoids the cumulative errors inherent in pipeline-based systems (OCR $\to$ Translation $\to$ Classification).
Complementarity of Objectives: The two-stage approach (generative followed by contrastive) outperforms either approach alone by 3–5 F1 points, suggesting that language modeling and metric learning objectives are complementary.
Importance of Regularization: The study highlights that different tasks within the same multimodal domain require vastly different regularization strategies; sentiment analysis, in particular, demanded aggressive regularization to prevent overfitting.
Cultural Nuance: While the system performs well, the author acknowledges limitations in handling culturally specific references and historical contexts that may not be present in the base model's pre-training data.

The author positions this work as a step toward more robust content moderation tools for low-resource languages, emphasizing the need for human review in deployment due to potential biases and false positives on legitimate political or cultural speech.