WhatsApp Vaccine Discourse (WhaVax): An Expert-Annotated Dataset and Benchmark for Health Misinformation Detection

Technical Summary: WhaVax – An Expert-Annotated Dataset and Benchmark for Health Misinformation Detection

Problem Statement
Health misinformation on encrypted messaging platforms like WhatsApp poses a significant public health challenge, particularly in Brazil, where declining vaccination rates threaten herd immunity. Unlike public social media, WhatsApp's end-to-end encryption and semi-private group structures hinder external monitoring and moderation, creating a "black box" for researchers. Existing datasets for vaccine misinformation are predominantly sourced from public platforms like Twitter/X, failing to capture the unique linguistic, structural, and contextual dynamics of private group conversations. Furthermore, there is a scarcity of high-quality, expert-validated datasets in Portuguese that reflect authentic conversational environments, limiting the development and evaluation of computational detection models under realistic data-scarcity constraints.

Methodology
The authors present WhaVax, a new dataset constructed through a rigorous, multi-stage pipeline:

1. Data Collection: Messages were harvested from large Brazilian public WhatsApp groups (accessible via invitation links) between 2020 and 2023. The collection utilized an automated pipeline capturing text and metadata (sender, group, timestamp) while anonymizing personally identifiable information to comply with Brazilian data protection laws.
2. Filtering and Deduplication:
   • Keyword Filtering: A broad set of Portuguese vaccination-related terms (including slang and obfuscated spellings like "v4c1n4") was used to isolate health-related content from a larger political corpus.
   • Semantic Deduplication: To remove near-duplicate content, message embeddings were generated using SentenceTransformers. Messages with a cosine distance below 0.02 were clustered, retaining only the first instance. This reduced the dataset by 5.18%.
3. Expert Annotation:
   • Protocol: Four medical professionals conducted a progressive, iterative annotation process. It began with a pilot phase to align criteria, followed by rounds of 50, 300, and finally 950 randomly selected messages.
   • Criteria: Annotators classified messages as "misinformation" or "non-misinformation." Disagreements were resolved via majority vote (at least 3 out of 4).
   • Outcome: The final gold-standard corpus consists of 950 messages (approx. 30% misinformation, 70% non-misinformation) with a Fleiss' Kappa of 0.621, indicating substantial inter-annotator agreement.
4. Benchmarking: The authors evaluated three families of models under realistic data-scarcity conditions (5-fold stratified cross-validation, repeated 8 times):
   • Classical ML: SVM, Logistic Regression, Random Forest, MLP, and XGBoost, using embeddings from BERTimbau and Qwen (8B).
   • Fine-tuned Small Language Models (SLMs): BERTimbau, RoBERTa, BERTuguês, and BioBERTpt.
   • Large Language Models (LLMs): Zero-shot and few-shot In-Context Learning (ICL) using LLaMA 3.1/3.2, DeepSeek, Qwen, and proprietary GPT models (GPT-5.1/5.2).

Key Contributions

1. WhaVax Dataset: A novel, expert-annotated corpus of vaccine-related WhatsApp messages, representing one of the first medically validated resources for encrypted messaging environments.
2. Empirical Characterization: A detailed analysis revealing that misinformation messages are significantly longer, contain more emotional punctuation (exclamation/question marks), use more emojis, and employ more capitalization than non-misinformation. Lexically, misinformation relies on risk/fear narratives (e.g., "death," "side effects"), while non-misinformation aligns with institutional sources. The study also identifies a subset of ambiguous cases (approx. 8.8%) where expert consensus was difficult, reflecting the complexity of health discourse.
3. Systematic Benchmark: A comparative evaluation of classification strategies under resource constraints, demonstrating that strong embeddings and LLM approaches can perform competitively even with small datasets.

Results

• Classical Models: When paired with high-quality embeddings (specifically Qwen 8B), classical models achieved strong performance. Logistic Regression with Qwen embeddings reached the highest macro-F1 score of 0.791, outperforming BERTimbau-based embeddings (0.747). This suggests that representation quality often outweighs classifier complexity in this domain.
• Fine-tuned SLMs: Performance was heterogeneous and generally lower than classical models with strong embeddings (e.g., BERTimbau: 0.712, RoBERTa: 0.580). The authors attribute this to a mismatch between pre-training domains and the informal, noisy nature of WhatsApp text, as well as insufficient data for effective adaptation.
• LLMs (ICL): Zero- and few-shot LLMs demonstrated competitive performance without supervised tuning.
  • Open-source: LLaMA 3.1 (8B) and DeepSeek-V3.2 (685B) achieved macro-F1 scores near 0.74.
  • Proprietary: GPT-5.1 and GPT-5.2 achieved the highest scores among LLMs (0.786 and 0.780, respectively), statistically matching the best classical approach.
  • Error Analysis: Models struggled with highly emotional or politicized language that mimicked misinformation but was labeled as non-misinformation by experts (false positives), and with subtle, insinuating messages that fostered distrust without explicit false claims (false negatives).

Significance and Claims
The paper claims that constructing a domain-specific, expert-annotated dataset for encrypted messaging is feasible and yields a reliable gold-standard corpus. The study highlights that:

• Data Quality over Quantity: Careful filtering and expert review can produce a small but highly informative dataset that supports effective automatic detection.
• Representation is Key: Strong embeddings (like those from Qwen) can compensate for simpler classifier architectures, making them a robust baseline for misinformation detection in informal settings.
• LLM Potential: Zero- and few-shot LLMs offer a viable alternative for detection tasks where supervised fine-tuning is constrained by data availability, though performance varies significantly based on model architecture and domain alignment.
• Ambiguity: The dataset captures a meaningful layer of ambiguity in health discourse, challenging the binary nature of standard classification and underscoring the need for models sensitive to pragmatics and context.

The authors position WhaVax as a rare resource to support computational modeling and public health research in encrypted environments, bridging the gap between public and private platform analysis. They do not claim to solve the misinformation problem but provide the necessary infrastructure (data and benchmarks) to advance the field.