💻 computer science

MemeScouts@LT-EDI 2026: Asking the Right Questions -- Prompted Weak Supervision for Meme Hate Speech Detection

This paper proposes a prompted weak supervision framework using a quantized Qwen3-VLM to decompose meme hate speech detection into targeted question-based labeling functions, achieving top rankings in English, Chinese, and Hindi for the LT-EDI 2026 shared task by effectively addressing the challenges of multimodal, culturally nuanced, and multilingual hate speech.

Original authors: Ivo Bueno, Lea Hirlimann, Enkelejda Kasneci

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ivo Bueno, Lea Hirlimann, Enkelejda Kasneci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to spot a specific type of joke that is actually mean-spirited (hate speech) in a collection of internet memes. Memes are tricky because they are like double-layered puzzles: they have a picture and text, and they often use sarcasm, irony, or inside jokes that only make sense if you know the cultural context.

The authors of this paper tried to build a computer system to solve these puzzles, specifically looking for jokes that are homophobic or transphobic. Here is how they did it, explained simply:

The Problem: Asking the "Big Question" Didn't Work

The researchers first tried asking a super-smart AI (called a Vision-Language Model) a single, giant question: "Is this meme hateful?"

Think of this like asking a student to solve a complex math problem, write an essay, and draw a picture all in one single sentence. The AI often got confused. It tried to guess the target, the tone, the sarcasm, and the final verdict all at once, and it made mistakes. This is especially hard when the memes are in different languages (English, Hindi, and Chinese) because the "inside jokes" change depending on the culture.

The Solution: The "Interrogation Room" Approach

Instead of asking one big question, the team changed their strategy. They decided to break the problem down into a series of small, specific questions, like an interrogation or a checklist.

They created a "Prompted Weak Supervision" system. Imagine a detective (the AI) who doesn't just guess if a suspect is guilty. Instead, the detective asks 89 specific questions about the evidence:

"Is the joke targeting a specific group?"
"Is the tone sarcastic?"
"Is the text actually saying the opposite of what it seems?"
"Is the image showing a stereotype?"

The AI answers these questions with simple "Yes/No" or short categories. These answers aren't the final verdict; they are clues.

The Assembly Line

Once the AI answers all 89 questions for a meme, the team takes those answers and feeds them into a simpler, classic computer program (a Random Forest classifier). Think of this program as a judge who listens to all the clues from the detective.

The Detective (AI): "The joke is about gender identity, it's sarcastic, and it targets a specific person."
The Judge (Classifier): "Okay, based on those specific clues, I am 90% sure this is hate speech."

What They Found

The results were like a magic trick:

Better Accuracy: This "checklist" method was much better than just asking the AI one big question. It worked especially well for Hindi and Chinese memes, where the cultural nuances are very subtle.
The "Pruning" Step: At first, they had too many questions (some were repetitive or confusing). They used a "pruning" tool to cut out the bad questions, leaving only the most helpful clues. This made the system even sharper.
The Rankings: In the competition they entered, their system took 1st place for English, 2nd for Chinese, and 3rd for Hindi.

The Catch (Limitations)

The authors are honest about the flaws in their system:

Western Bias: The questions were mostly written from a Western perspective. They might miss specific cultural signs of hate speech in India or China that don't look like the "Western" version.
Reading Text: The AI has to read text inside the images. Sometimes it misses words or reads them wrong, which throws off the clues.

The Bottom Line

The paper shows that when trying to catch subtle, mean-spirited jokes in memes, it's better to break the problem down into small, manageable questions and let a system weigh the answers, rather than asking a super-computer to guess the whole answer in one go. It's the difference between asking a student "What is the answer?" versus asking them to show their work step-by-step.

1. Problem Statement

Detecting hate speech in memes is a complex challenge due to their multimodal nature (combining text and images) and reliance on culturally grounded cues such as sarcasm, irony, and shared knowledge.

Limitations of Current Approaches: Direct end-to-end prompting of Vision-Language Models (VLMs) is often brittle. A single prediction must simultaneously resolve target identification, stance, implicitness, and irony, which is difficult in multilingual settings.
Multilingual & Cultural Gaps: Standard models struggle with subtle cultural nuances, slang, and specific expressions of homophobia and transphobia across different languages (English, Hindi, Chinese).
Cost: Fine-tuning large models for specific tasks incurs high computational and data costs.

2. Methodology: Prompted Weak Supervision (PWS)

The authors propose a Prompted Weak Supervision (PWS) framework that decomposes the complex task of meme classification into structured, question-based labeling functions (LFs). Instead of a single end-to-end label, the system generates structured features for a downstream classifier.

Core Pipeline Components:

Question Generation (Labeling Functions):
- The team constructed 59 initial questions (later expanded to 89) designed to capture complementary aspects of hate speech (e.g., target identification, explicit/implicit hate, narrator identity, stance).
- Answers are constrained to binary, ordinal, or categorical options to ensure structured output.
Feature Extraction:
- A quantized Qwen3-VLM (Qwen3-VL-30B-A3B-Instruct, nf4-quantized) is used to answer these targeted questions for each meme.
- The model receives the meme image and the specific LF question.
- Responses are mapped to integers, creating a feature vector for each meme.
Classification:
- A lightweight Random Forest classifier (500 estimators) is trained on these feature vectors for each language.
- The model uses balanced class weights to handle imbalanced datasets.
Iterative Refinement:
- AddLF (Label Function Expansion): The team analyzed misclassified examples to identify recurring patterns (e.g., narrator identity) not captured by initial LFs, adding 30 new questions.
- Pruning Strategies: Two methods were employed to reduce redundancy and noise:
  - F1Prune: Greedily removes features one by one if validation Macro-F1 improves.
  - ImpPrune: Removes the top $k$ least important features based on Random Forest feature importance scores.

3. Key Contributions

Novel Framework: Introduction of a PWS approach for multimodal hate speech detection that replaces brittle end-to-end prompting with structured, interpretable question-based features.
Multilingual Performance: The method significantly outperforms direct VLM classification, particularly in low-resource or culturally distinct languages (Chinese and Hindi).
Interpretability: By analyzing feature importance and clustering (via UMAP), the authors provide insights into why the model makes decisions, revealing redundancy and complementarity among labeling functions.
State-of-the-Art Results: The system achieved top rankings in the LT-EDI 2026 shared task (1st in English, 2nd in Chinese, 3rd in Hindi).

4. Experimental Results

The system was evaluated on the LT-EDI 2026 Homophobia and Transphobia Meme Classification dataset (English, Hindi, Chinese).

Method	English (Macro-F1)	Chinese (Macro-F1)	Hindi (Macro-F1)	All (Aggregated)
Direct Qwen3-VL	0.77	0.32	0.21	0.13
Direct w/ Reasoning	0.67	0.10	0.08	0.10
Base PWS Model	0.85	0.66	0.64	0.47
AddLF + ImpPrune	0.85	0.72	0.67	0.44

Performance Gap: The PWS approach significantly outperformed direct VLM prompting. The gap was most pronounced in Chinese and Hindi, where direct prompting struggled (0.32 and 0.21 vs. 0.72 and 0.67).
Refinement Impact:
- AddLF consistently improved or maintained performance, especially for non-English languages.
- ImpPrune was more effective than F1Prune for per-language performance, reducing features from 89 to 33 for Hindi while improving scores, indicating many initial LFs were noisy or redundant.
Cross-Lingual Analysis:
- F1Prune removed mostly language-specific features (low Jaccard similarity across languages).
- ImpPrune removed a large set of shared, uninformative features (high Jaccard similarity), suggesting the presence of language-agnostic "weak signals" that hindered performance.

5. Significance and Insights

RQ1 (Effectiveness): Prompted weak supervision is a superior strategy for meme hate speech detection, offering robustness against the brittleness of single-shot VLM prompts.
RQ2 (Language/Culture): While the method improves multilingual performance, the analysis revealed that English-centric LFs (reflecting Western perspectives on queer issues) may not transfer perfectly. However, pruning helps isolate language-specific signals.
RQ3 (Model Behavior): Feature analysis showed that a small subset of LFs carries the majority of the predictive weight. Some features are redundant (e.g., asking about sexual orientation in slightly different ways), while others provide complementary information (e.g., distinguishing neutral stances).
Limitations & Future Work:
- Cultural Bias: LFs were primarily designed from a Western perspective, potentially missing culturally specific markers in Hindi and Chinese.
- OCR: The lack of a dedicated OCR stage means text embedded in images is processed jointly with the image, potentially introducing errors.
- Future Directions: Incorporating explicit OCR, culturally adaptive prompting, and diverse perspectives in LF design to improve generalizability without expensive retraining.

Conclusion

The paper demonstrates that decomposing complex multimodal tasks into targeted questions allows for more robust and interpretable hate speech detection. By leveraging a quantized VLM for feature extraction and a lightweight classifier for aggregation, the authors achieved state-of-the-art results in a multilingual shared task, proving that "asking the right questions" is more effective than "guessing the right answer" in a single shot.