CHASM: Unveiling Covert Advertisements on Chinese Social Media

1. Problem Statement

The paper addresses the critical and under-explored threat of covert advertisements on social media platforms. Unlike traditional ads, covert advertisements are deliberately designed to mimic regular user-generated content (e.g., product experience sharing, lifestyle posts) to deceive consumers into making purchases without explicitly disclosing their commercial nature.

Key Challenges:

• Deceptive Nature: They blend seamlessly with organic content, making them difficult to distinguish from genuine user reviews.
• Multimodal Complexity: Detection requires analyzing text, images, and comments simultaneously, as promotional cues are often hidden in specific modalities (e.g., links in comments, brand logos in images).
• Subjectivity: The line between a genuine product recommendation and a paid promotion is often blurry, leading to annotation inconsistencies.
• Lack of Resources: Prior to this work, there were no dedicated datasets or benchmarks for training Multimodal Large Language Models (MLLMs) to detect this specific threat.

2. Methodology

A. Dataset Construction: CHASM

The authors introduced CHASM (Covert Hype Advertisement in Social Media), the first high-quality, manually curated dataset for this task.

• Source: Data was collected from RedNote (Xiaohongshu), a major Chinese lifestyle and shopping platform with over 120 million daily active users.
• Scale: The dataset contains 4,992 instances, comprising 612 covert advertisements (12.3%) and 4,380 non-covert samples (87.7%).
• Privacy & Ethics: Strict anonymization was applied (masking PII in text, blurring faces in images). Data collection adhered to platform privacy policies, and annotators were paid above minimum wage.
• Annotation Process:
  • Guidelines: Developed evidence-based criteria focusing on promotional intent, language style (clickbait vs. casual), and structural patterns (single brand focus vs. diverse sharing).
  • Quality Control: A dynamic workflow was used where simple cases were labeled by one annotator, while ambiguous "product-sharing" cases underwent a three-annotator majority voting mechanism.
  • Gold Standard: 50 gold-standard questions were used to monitor annotator accuracy, achieving 94% accuracy on these tests.

B. Evaluation Framework

The authors evaluated a wide range of Multimodal Large Language Models (MLLMs) using three distinct settings:

1. Zero-Shot Prompting: Models were given the post content and a brief instruction to classify.
2. In-Context Learning (ICL): Models were provided with few-shot examples of both covert and non-covert posts.
3. Fine-Tuning: Open-source models were fine-tuned on the CHASM dataset using LoRA adapters.

Models Tested:

• Open-Source: InternVL2.5, LLaVA, Qwen2.5-7B, DeepSeek-VL2, Llama-4.
• Proprietary/Closed-Source: GPT-4o, DeepSeek-V3, Qwen-Max, GLM-4 series, Gemini 2.0/2.5 Pro, and reasoning-specific models (QvQ-Max, Step-R1-V-Mini).

C. Error Analysis

The authors categorized model failures into four types:

1. Insufficient Evidence: Misclassifying normal posts as ads without clear promotional cues.
2. Missing Clue: Failing to detect hidden links in comments or visual cues in images.
3. Textual Style: Misinterpreting the tone (e.g., failing to distinguish clickbait from genuine enthusiasm).
4. Structural Pattern: Missing the distinction between posts focusing on a single brand (likely ads) vs. multiple brands (likely genuine sharing).

3. Key Contributions

1. New Task Definition: Formally defined the task of covert advertisement detection, highlighting its unique multimodal and deceptive characteristics.
2. CHASM Dataset: Released a high-quality, privacy-preserving dataset of nearly 5,000 real-world instances, including challenging "product-sharing" negatives that closely resemble ads.
3. Comprehensive Benchmarking: Provided the first systematic evaluation of state-of-the-art MLLMs (including reasoning models) on this task.
4. Empirical Insights: Demonstrated that while current models struggle in zero-shot settings, fine-tuning significantly improves performance, though significant gaps remain.

4. Experimental Results

• Zero-Shot & In-Context Performance:

  • Current MLLMs are not sufficiently reliable for this task.
  • The best zero-shot performer, GPT-4o, achieved an F1-Score of only 59.7%.
  • Even advanced reasoning models (e.g., Gemini 2.5 Pro) did not significantly outperform standard models, suggesting that reasoning capabilities alone do not solve the nuance of covert detection.
  • Precision was generally low, indicating models tend to over-flag normal content as advertisements.

• Fine-Tuning Results:

  • Fine-tuning open-source models yielded substantial improvements.
  • Qwen2.5-7B, after fine-tuning, achieved an F1-Score of 75.6%, significantly surpassing the zero-shot GPT-4o baseline.
  • This proves the dataset is effective for training models to align with human judgment.

• Modality Analysis:

  • Removing images or comments from the input significantly degraded performance. This confirms that covert advertisements rely heavily on cross-modal cues (e.g., text in comments directing to a link in an image).

• Error Analysis Findings:

  • Fine-tuning drastically reduced "Insufficient Evidence" errors (misclassifying normal posts).
  • However, even fine-tuned models struggled with structural patterns and subtle clues in comments, indicating these remain the hardest aspects of the task.

5. Significance and Future Directions

Significance:

• Regulatory Impact: Covert advertising violates laws in China, the US, and other regions. This work provides the technical foundation for automated compliance and consumer protection.
• Platform Moderation: Offers a cost-effective path (via fine-tuned open-source models) for platforms to detect deceptive content that currently evades filters.
• Research Gap: Fills a critical void in social media moderation research, which has historically focused on hate speech or fake news but neglected commercial deception.

Future Directions:

• Dynamic Detection: Moving beyond static post analysis to monitor evolving comment sections where ads often appear later.
• User Behavior Integration: Incorporating signals like user reports, viewing duration, and creator history (subject to privacy constraints) to improve detection.
• Cross-Platform Expansion: Extending the methodology to other platforms (e.g., Douyin, Weibo) and languages.

In conclusion, the paper establishes that while current MLLMs cannot reliably detect covert advertisements without training, the CHASM dataset enables significant performance gains through fine-tuning, offering a viable path toward more honest and fair social media environments.