Assessing Capabilities of Large Language Models in Social Media Analytics: A Multi-task Quest

1. Problem Statement

Social media analytics faces significant challenges due to the complexity, informality, and rapid evolution of user-generated content (e.g., slang, memes, short text). While Large Language Models (LLMs) have shown promise in various NLP tasks, their specific capabilities and limitations in the context of social media remain under-evaluated. Existing studies often focus on single tasks or lack rigorous controls for data leakage (models trained on data they are tested on) and generalization to unseen time periods.

The paper addresses three core tasks essential for understanding user identity in social media:

1. Social Media Authorship Verification: Determining if a specific post was written by a target user.
2. Social Media Post Generation: Generating content that authentically mimics a specific user's style and preferences.
3. User Attribute Inference: Predicting latent user attributes (occupation and interests) from textual content.

2. Methodology

The study utilizes a comprehensive framework evaluated on a large Twitter (X) dataset (120k+ users, 15-week period). The methodology is divided into three distinct experimental phases:

A. Social Media Authorship Verification (Task I)

• Objective: Binary classification (Target User vs. Other).
• Sampling Strategy: To ensure robustness, the authors introduced a systematic sampling framework:
  • Positive Users: Sampled via Random, Recent Active, and Top Active strategies.
  • Negative Posts: Sampled via Random, Topic-similar (semantic similarity), and Social Graph-based (Followers, Followees, Reciprocal) strategies to create realistic "impostor" challenges.
• Unseen Data Evaluation: To mitigate "seen data" bias, models were tested on tweets collected from January 2024 onward, well beyond the knowledge cutoff dates of the evaluated LLMs (e.g., GPT-4 cutoff: Oct 2023).
• Metrics: Weighted F1-score.

B. Social Media Post Generation (Task II)

• Objective: Generate synthetic tweets conditioned on a user's profile (bio, stats) and historical posts (few-shot examples).
• Dataset: Restricted to VAPOR users (Verified, Active, Profile-rich, Original) to ensure high-quality ground truth.
• Evaluation Metrics:
  • Lexical/Semantic: BLEU, ROUGE-1, ROUGE-L, and Perplexity (using GPT-2).
  • Proposed Novel Metrics:
    • Average Gen vs. Real (AGR) & Average Real vs. Gen (ARG): Mean maximum cosine similarity between generated and real posts.
    • Dispersion Ratios (GDR/RDR): Measures the breadth of coverage (how many unique real posts are matched by generated ones).
• User Study: A human evaluation involving 12 valid participants who rated the "authenticity" of LLM-generated tweets against their own real tweets on a 5-point scale.

C. User Attribute Inference (Task III)

• Objective: Predict user occupations and interests.
• Taxonomies:
  • Interests: IAB Tech Lab Content Taxonomy v3.1 (Top-level categories).
  • Occupations: 2018 U.S. Standard Occupational Classification (SOC) at Level 1 (18 groups) and Level 2 (38 groups).
• Approach: LLMs were prompted to classify users based solely on 50 sampled tweets, without access to profile bios or network data, to test pure textual inference capabilities.
• Metrics: Accuracy and Weighted F1-score.

3. Key Contributions

1. Unified Benchmarking Framework: The first study to jointly evaluate multiple state-of-the-art LLMs (GPT-4, GPT-4o, GPT-3.5, Gemini 1.5 Pro, DeepSeek-V3, Llama 3.2, BERT) across these three fundamental social media tasks under a single, reproducible framework.
2. Mitigation of Data Bias: Introduction of a "Unseen Data" evaluation protocol using post-cutoff tweets (Jan 2024+) to strictly test generalization and prevent data leakage.
3. Novel Generation Metrics: Development of dispersion-based metrics (GDR/RDR) and a structured user study to bridge the gap between automatic metrics and human perception of authenticity.
4. Standardized Taxonomies: Grounding attribute inference in established, hierarchical taxonomies (IAB and SOC) to enable fine-grained, reproducible labeling and comparison with real-world standards.

4. Key Results

Authorship Verification (Task I)

• Top Performer: GPT-4 achieved the highest average F1-score (0.845) and maintained strong performance (0.85) on unseen data.
• Second Tier: Gemini and DeepSeek showed robust performance (~0.75–0.79).
• Baselines: Traditional models (Random Forest, TF-IDF) and smaller LLMs (Llama, BERT) performed significantly worse or showed high variance.
• Insight: Negative post sampling strategy (specifically Topic-similar and Graph-based) had a larger impact on accuracy (approx. 17% variance) than positive user sampling.

Post Generation (Task II)

• Semantic Alignment: DeepSeek achieved the highest semantic similarity (AOS) but suffered from high perplexity (lower fluency).
• Lexical Reuse: Llama achieved the highest BLEU/ROUGE scores, indicating strong lexical copying but lower semantic diversity.
• Balance: GPT-4o offered the most balanced profile, combining strong semantic similarity with the lowest perplexity among LLMs.
• Human Perception: In the user study, Gemini and Llama received the highest "authenticity" ratings (Mean ~3.95/5), suggesting that lexical reuse and paraphrasing can sometimes fool humans better than deep semantic alignment.

User Attribute Inference (Task III)

• Top Performer: Gemini outperformed all other models across interests and both occupation levels (L1 and L2).
• Granularity Impact: Performance dropped significantly as classification granularity increased (L1 to L2), highlighting the difficulty of fine-grained inference.
• Underperformance: Llama consistently underperformed, particularly in fine-grained occupation tasks.

5. Significance and Implications

• Benchmarking Standard: This work establishes a reproducible, multi-task benchmark for social media analytics, moving beyond single-task evaluations.
• Generalization Proof: The study demonstrates that while LLMs are powerful, their performance varies significantly based on the task (verification vs. generation vs. inference) and the specific model architecture.
• Human vs. Machine Alignment: The user study reveals a disconnect between automatic metrics (like BLEU) and human perception of authenticity, suggesting that future evaluation must incorporate human-in-the-loop validation.
• Ethical Considerations: The authors highlight the risks of LLM-driven identity modeling, including impersonation and privacy-sensitive profiling, emphasizing that these tools should be used for benchmarking rather than unsafeguarded deployment.

Conclusion: The paper concludes that GPT-4 is superior for authorship verification, DeepSeek excels in semantic alignment for generation, and Gemini is the most robust for attribute inference. However, no single model dominates all tasks, and the choice of model depends heavily on the specific social media analytics goal.