Technical Summary: Detecting AI-Generated Content on Social Media with Multi-modal Language Models

Problem Statement

The rapid advancement of generative AI has enabled the creation of photorealistic images and videos that are increasingly disseminated on social media. These AI-generated contents (AIGC) are frequently exploited for spam, misinformation, manipulation, harassment, and fraud. Existing detection methods face three critical limitations:

1. Poor Generalization: Models trained on static datasets often fail to detect content generated by newer, evolving models.
2. Single-Modality Reliance: Many approaches rely solely on visual artifacts (e.g., frequency analysis or GAN-specific cues) and ignore rich multi-modal signals available in social media posts (text, comments, context).
3. Lack of Interpretability: Most detectors provide binary outputs without human-readable explanations, making their judgments difficult to interpret and trust.

Methodology

The authors propose a unified pipeline that continuously curates diverse multi-modal data and trains a compact vision-language model (VLM) for both detection and explanation.

1. Model Architecture

The system utilizes a LLaVA-based architecture comprising three components:

• Vision Encoder: A Perception Encoder (300M parameters) chosen for its strong zero-shot classification capabilities. Images are cropped to 448×448 and processed into 256 visual tokens via adaptive pooling.
• Language Model: Llama3.2-3B-Instruct is selected to ensure high inference efficiency while maintaining sufficient capacity for reasoning.
• Projection Layer: A single linear layer with zero-weight initialization connects the visual tokens to the language model.

2. Data Curation Pipeline

The training data is curated through a continuous pipeline involving three distinct stages:

• Stage 1: Image Captioning (60M+ samples): Data is sourced from Shutterstock (cleaned and captioned by Llama-3.2-90B) and social media platforms (captioned by Llama4-Scout). A subset of 164K high-quality images undergoes human annotation and refinement to align image-text modalities.
• Stage 2: Multi-task Fine-tuning (130K+ samples): The model is fine-tuned on tasks such as named entity recognition, keyword generation, hashtag generation, and genre classification. This stage bootstraps the model with initial knowledge of AIGC signals often present in generated keywords or hashtags.
• Stage 3: AIGC Fine-tuning (200K+ samples): Data is collected from social media and labeled using heuristics (e.g., explicit mentions of "AI-generated," "Midjourney," or model-predicted AIGC keywords). The dataset consists of 30K positive (AIGC) and 170K negative samples, with explanations annotated by Llama4-Maverik. Crucially, during this stage, 90% of text information is randomly dropped to force the model to learn from visual features rather than relying solely on textual clues.

3. Training Recipe

The model, named IFM-AIGCSPOTTER-3B, undergoes a three-stage post-training process:

1. Image Captioning: Aligns text-image modalities, initially freezing the LM and Vision Encoder to train the projection layer, followed by end-to-end training.
2. Multi-task Fine-tuning: Enhances multimodal content understanding.
3. AIGC Detection Fine-tuning: Focuses exclusively on detection and explanation capabilities using the curated AIGC dataset.

Key Results

Benchmark Performance

Evaluated on public datasets FakeClue (in-distribution) and LOKI (out-of-distribution):

• IFM-AIGCSPOTTER-3B achieved state-of-the-art performance among fine-tuned models, reaching 98.6% accuracy on FakeClue and 83.9% accuracy on LOKI.
• The model demonstrated robust generalization, particularly for images containing persons, animals, and scenes.
• Trade-off Observation: The authors noted that incorporating explanation generation during training slightly reduced detection accuracy compared to detection-only training, suggesting a trade-off between explainability and pure performance.

Internal Social Media Evaluation

On internal datasets across Facebook, Instagram, and Threads:

• The model achieved an overall Precision of 88.6% and Recall of 85.3%.
• Ablation Studies revealed that:
  • Random text dropout was crucial for recall; without it, the model over-relied on textual cues (recall dropped to 50%).
  • The Perception Encoder offered a slight precision advantage over CLIP-based encoders.
  • Extending inputs to multi-frame videos (4 or 8 frames) significantly improved performance over single-frame inputs.

Generalization to "In-the-Wild" Benchmarks

On the Chameleon dataset (containing highly realistic AIGC), the model achieved 86.0% accuracy, outperforming 10 existing baselines by a margin of over 20% without dedicated adaptation.

Production Deployment

The model was deployed on 4 Nvidia H100 nodes to screen posts with >500 views.

• Impact: A 6-week A/B test showed a statistically significant increase in views for marginal users (+0.21%) and a +0.22% gain in Young-Adult session time, indicating that effective AIGC detection improves user engagement and experience.

Significance and Claims

The paper claims that it is feasible to perform effective AIGC detection in dynamic, real-world social media environments by leveraging a unified vision-language model trained on continuously curated, diverse multi-modal data.

Key contributions include:

1. Continuous Adaptation: A pipeline that allows for the continuous curation of new high-quality AIGC data, addressing the issue of distribution shift from evolving generative models.
2. Multi-modal Reasoning: Moving beyond single-modality detection to utilize visual, textual, and contextual signals simultaneously.
3. Explainability: Providing human-readable explanations (analyzing semantic, stylistic, and local visual artifacts) alongside detection, which aids in interpretability and user trust.
4. Practical Viability: Demonstrating that a compact (3B parameter) model can achieve state-of-the-art performance while being deployable at scale, resulting in tangible downstream benefits for user engagement.

Limitations

The authors acknowledge several limitations:

• Reactive Nature: While the pipeline mitigates distribution shift, it remains reactive to emerging generators and may lag behind models specifically designed to evade detection.
• Computational Cost: Large-scale deployment still incurs nontrivial computational costs.
• Explanation Faithfulness: While explanations improve interpretability, they do not guarantee faithful causal attribution and may occasionally reflect model biases or hallucinations (e.g., hallucinating "unnatural textures" in artifact-free regions).