💬 NLP

mdok-style at SemEval-2026 Task 10: Finetuning LLMs for Conspiracy Detection

The paper presents the mdok-style system, which achieves competitive results (8th out of 52) in SemEval-2026 Task 10's conspiracy detection challenge by finetuning the Qwen3-32B model with data augmentation and self-training to address limited training data.

Original authors: Dominik Macko

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dominik Macko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to spot a specific type of rumor in a massive, noisy crowd of people chatting on a social media platform (Reddit). Some people are just sharing facts, while others are spreading wild conspiracy theories. Your job is to build a robot detective that can listen to a single comment and shout, "Yes, that's a conspiracy!" or "No, that's just normal talk."

This paper describes how the mdok-style team built their robot detective for a competition called SemEval-2026 Task 10. Here is how they did it, explained simply:

1. The Problem: A Tiny Library for a Giant Brain

The team had a big problem: they had a very smart robot (a Large Language Model called Qwen3-32B, which is like a super-brain with 32 billion "neurons"), but they only had a tiny library of training books (about 4,000 examples of comments).

Usually, teaching a super-brain requires a massive library. With so few examples, the robot might get confused or "overthink" things. It's like trying to teach a master chef to cook a complex dish using only three recipes.

2. The Solution: The "Magic Copy Machine" and the "Student-Teacher" Loop

To fix the small library problem, the team used two clever tricks borrowed from their previous work on spotting AI-generated text.

Trick A: The Magic Copy Machine (Data Augmentation)
Instead of just copying the existing comments, they used a "magic copy machine" to create slightly different versions of the same stories. They didn't just copy-paste; they:

Changed the case: They wrote some comments in ALL CAPS and some in all lowercase. They figured out that a conspiracy theory is still a conspiracy theory whether it's shouted or whispered.
Hid the identities: They replaced specific names, emails, and phone numbers with generic tags like [USER] or [PHONE]. This taught the robot to focus on the idea of the conspiracy, not on specific people.
Swapped the letters: They swapped some letters for look-alikes from other alphabets (like swapping an 'a' for a Greek 'alpha'). This is like wearing a disguise. By training the robot to see through these disguises, they made it much harder for the robot to be tricked by visual tricks later on.

They didn't use all these copies, just a small slice (10%) of them, to keep the training balanced.

Trick B: The Student-Teacher Loop (Self-Training)
This is the most creative part. Since they didn't have labels for the test data (the final exam questions), they let the robot teach itself.

The Teacher: First, they trained the robot on the small, known library.
The Exam: They asked the robot to guess the answers for the unknown test comments.
The Filter: The robot was very confident about some answers (e.g., "I am 99% sure this is a conspiracy"). The team took only those super-confident guesses and treated them as "gold standard" new training data.
The Re-Lesson: They fed these new "gold" examples back to the robot and taught it again. This is like a student taking a practice test, keeping only the questions they got 100% right, and then studying those again to become even smarter.

3. The Result: A Top-Tier Detective

The team entered their robot into the competition.

The Score: Their robot achieved a score of 0.78 (on a scale where 1.0 is perfect).
The Ranking: Out of 52 teams, they finished 8th. This puts them in the top 20% (the 85th percentile).

Interestingly, they found that even a much smaller, simpler robot (a model called DeBERTa) did almost as well as their giant super-brain. This suggests that while their fancy "magic copy machine" and "self-teaching" tricks worked great, sometimes a smaller, cheaper tool can do the job almost as effectively.

Summary

The team took a massive, powerful AI model and taught it to spot conspiracy theories in social media comments. Because they didn't have enough data, they used disguises (changing text formats) to create more practice material and let the AI teach itself using its own confident guesses. The result was a highly competitive system that ranked 8th out of 52, proving that techniques originally designed to spot AI-generated text can also be used to spot human-made conspiracy theories.

Technical Summary: mdok-style at SemEval-2026 Task 10

Problem Definition
The paper addresses SemEval-2026 Task 10 (PsyCoMark), a shared task focused on detecting conspiracy beliefs within social media discourse. Specifically, the authors tackle Subtask 2, a binary classification problem requiring the identification of whether an English Reddit comment expresses a conspiracy belief (Yes/No). The primary challenge highlighted is the scarcity of labeled training data, consisting of only 1,715 positive and 2,263 negative samples, with text lengths filtered between 160 and 1,000 characters.

Methodology
The proposed system, named mdok-style, adapts a robust machine-generated text detection framework (originally developed for PAN@CLEF2025) to the domain of conspiracy detection. The approach relies on three core components:

Base Model and Fine-tuning: The system utilizes the Qwen3-32B large language model (LLM), selected from a comparison of Qwen3 and Gemma-3 families. The model is fine-tuned for binary sequence classification using QLoRA (4-bit quantization) for parameter efficiency. The optimization process employs the paged AdamW optimizer with a cosine learning rate schedule (2e-5) and a warmup ratio of 0.03.
Data Augmentation: To mitigate the small dataset size, the authors employ four specific augmentation techniques applied to 10% of the training data copies:
- Anonymization: Replacing regex-identified emails, user mentions, and phone numbers with tags ([EMAIL], [USER], [PHONE]), while retaining the original URL replacement ([URL]).
- Case Variation: Generating both lowercase and uppercase versions of the text to enforce case insensitivity and focus on semantic meaning.
- Homoglyphication: Introducing visually similar character swaps from different scripts to increase robustness against tokenization-based classifier confusion.
- De-duplication: Removing redundant unmodified copies from the final training set.
Self-Training: The system implements an iterative semi-supervised learning approach. An initial model is trained on the augmented gold-labeled data. It then predicts labels for the unlabeled development and test sets. Only predictions with extreme confidence (probability $\ge$ 0.99 for "Yes" or $\le$ 0.01 for "No") are retained as "silver labels." These high-confidence pseudo-labels are combined with the original training data to retrain the detector, aiming to expand the effective training set without propagating significant errors.

Key Contributions

Cross-Domain Transfer: The paper demonstrates the successful transfer of the mdok framework, originally designed for machine-generated text detection, to the distinct domain of conspiracy theory detection.
Robustness via Augmentation: The study validates that combining specific text obfuscation techniques (homoglyphs, anonymization) with self-training can effectively address data scarcity in LLM fine-tuning.
System Implementation: The authors provide a full open-source implementation (via the IMGTB framework) and a detailed replication guide, including the specific hyperparameters and data processing pipelines used.

Results
The submitted system achieved a Macro F1 score of 0.78 on the official test set.

Ranking: The system placed 8th out of 52 submissions, ranking in the 85th percentile.
Comparative Performance: While the Qwen3-32B with self-training and a shifted classification threshold (0.7) yielded the highest score, the authors note that smaller models, such as DeBERTa-Large (<0.5B parameters), achieved a competitive Macro F1 of 0.75, suggesting a favorable trade-off between performance and computational cost.
Ablation: The inclusion of self-training and data augmentation consistently improved performance over the base model without these components (e.g., Qwen3-32B without self-training scored 0.76).

Significance and Claims
The paper modestly claims that the mdok-style approach, originating from machine-generated text detection, is a viable and competitive strategy for conspiracy detection. The authors emphasize that their method successfully leverages data augmentation and self-training to overcome the limitations of small training datasets. They acknowledge that while their system is highly competitive, the computational cost of 32B parameter models must be weighed against the marginal gains over smaller, more efficient models like DeBERTa. The work concludes that the approach is effective but notes limitations regarding the evaluation of only English texts and a limited exploration of base model architectures.

1. The Problem: A Tiny Library for a Giant Brain

2. The Solution: The "Magic Copy Machine" and the "Student-Teacher" Loop

3. The Result: A Top-Tier Detective

Summary

Technical Summary: mdok-style at SemEval-2026 Task 10

More like this