💻 computer science

Wisdom of the LLM Crowd: A Large Scale Benchmark of Multi-Label U.S. Election-Related Harmful Social Media Content

This paper introduces USE24-XD, a large-scale, multi-label dataset of nearly 100k U.S. election-related social media posts annotated by a "wisdom-of-the-crowd" of six LLMs and validated against human raters to enable scalable detection of harmful content while revealing systematic subjectivity in labeling behaviors.

Original authors: Qile Wang, Prerana Khatiwada, Carolina Coimbra Vieira, Benjamin E. Bagozzi, Kenneth E. Barner, Matthew Louis Mauriello

Published 2026-02-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qile Wang, Prerana Khatiwada, Carolina Coimbra Vieira, Benjamin E. Bagozzi, Kenneth E. Barner, Matthew Louis Mauriello

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet during an election is like a massive, chaotic town square. Everyone is shouting, posting signs, and spreading stories. Some stories are true, some are jokes, some are scary rumors, and some are outright lies designed to make people angry or afraid.

The researchers in this paper wanted to build a super-smart "filter" to sort through this noise and find the dangerous stuff (like lies, hate speech, or scary rumors) before it spreads too far. But sorting through 100,000 posts by hand would take a team of people years to do. So, they asked a new question: Can we use Artificial Intelligence (AI) to do the sorting for us?

Here is the story of how they tried to answer that, explained simply:

1. The Problem: The "Needle in a Haystack"

The team collected nearly 100,000 posts from X (formerly Twitter) during the 2024 U.S. election. They wanted to tag them into five specific "buckets" of bad content:

Conspiracy: "The election was stolen by aliens!" (Secret plots).
Sensationalism: "BREAKING: The world is ending tomorrow!" (Scary, over-the-top fear).
Speculation: "I bet the candidate will drop out next week." (Wild guesses without proof).
Hate Speech: Attacking people based on who they are.
Satire: Funny jokes that look like news but aren't (like The Onion).

2. The Experiment: The AI "Book Club"

Instead of hiring one person to read every post, the researchers hired six different AI models (think of them as six different super-smart book club members with different personalities).

They gave each AI the same 1,000 posts and asked: "Does this post fit into any of our five bad-content buckets?"
They also hired 34 real humans (via a crowdsourcing website) to do the same job, just to see how the AI compared to real people.

3. The Big Surprise: The AI "Crowd" Was More Consistent

Usually, we think humans are the gold standard for judging things like "is this a joke?" or "is this hate speech?" because humans understand nuance. But the researchers found something interesting:

The Humans were Messy: When different humans looked at the same post, they often disagreed. One person might think a post was a "joke" (Satire), while another thought it was "hate speech." This is because humans have their own political biases and backgrounds.
The AI was Steady: The six AIs agreed with each other much more often than the humans did. They were like a well-oiled machine.
The "Wisdom of the AI Crowd": The researchers realized that if they took the majority vote of the three best AIs, they got a result that was even better than any single AI or any single human. It was like asking three experts for their opinion and going with the answer two of them agreed on.

4. The Human Factor: Politics Changes How We See Things

The study also looked at why the humans disagreed. They found that who you are changes what you see.

If a human annotator was very conservative, they were more likely to label a post as "Hate Speech."
If they were a "Centrist" or independent, they were more likely to label it as "Speculation" (just a wild guess).
The Metaphor: Imagine looking at a painting through different colored glasses. A person wearing red glasses sees the whole painting as red; a person wearing blue glasses sees it as blue. The AI didn't have colored glasses, so it saw the "shape" of the content more consistently, even if it missed some of the subtle human context.

5. The Result: A New Tool for the Future

The team created a massive, clean dataset called USE24-XD.

What they found: About 60% of the election posts they looked at had at least one "bad" label. This means harmful content isn't a rare glitch; it's a huge part of the conversation.
The Win: They proved that using a "team" of AIs is a cheap, fast, and surprisingly accurate way to clean up social media. It's not perfect (it sometimes misses subtle jokes), but it's way faster and cheaper than hiring armies of humans.

The Takeaway

Think of this study as building a new kind of sieve.
In the past, we tried to sift through the internet's noise with tiny, hand-made sieves (humans), which was slow and inconsistent. This paper shows that we can build a giant, automated sieve made of AI. While the AI sieve isn't perfect, if you use a few different sieves together and take the majority vote, you catch almost all the "bad apples" without needing to hire a thousand people to pick through the fruit basket.

This gives researchers and platforms a powerful new tool to understand and fight misinformation before it causes real-world damage.

1. Problem Statement

The rapid spread of election-related misinformation and harmful political content on social media undermines democratic integrity. While detecting such content is crucial, existing efforts face significant challenges:

Data Scarcity & Quality: There is a lack of large-scale, standardized, and rigorously curated datasets specifically for election-related harms. Existing datasets often rely on heuristic labeling (e.g., keywords) or focus narrowly on "fake news," missing nuanced categories like satire or speculation.
Annotation Costs: Manual annotation of unstructured social media data is expensive, time-consuming, and prone to inconsistency, particularly for subjective categories.
LLM Evaluation Gap: While Large Language Models (LLMs) are increasingly used for annotation, there is limited systematic comparison of their performance across diverse harmful content categories, nor is there a clear understanding of how to best aggregate their predictions (e.g., single model vs. ensemble) compared to human annotators.

2. Methodology

The study introduces USE24-XD, a large-scale dataset and a novel annotation framework combining LLMs and crowdsourcing.

A. Dataset Collection (USE24-XD)

Source: ~100,000 posts collected from X (formerly Twitter) between October 17, 2024, and July 16, 2025, covering the U.S. presidential election cycle.
Filtering: Posts were filtered to include only original content containing "election" and "2024." Duplicates and posts with <5 words were removed.
Metadata: Enriched with spatio-temporal data (user location, though user-reported), engagement metrics, and sentiment analysis.
Final Size: 97,696 cleaned posts.

B. Annotation Framework

The study defines five multi-label binary categories of harmful content:

Conspiracy: Exaggerated claims of secret plots or stolen elections.
Sensationalism: Alarmist, dramatic, or fear-inducing language.
Hate Speech: Personal attacks, slurs, or incitement of violence based on identity.
Speculation: Unverified assertions or predictions lacking credible evidence.
Satire: Humor or irony used to critique politics (often ambiguous).

C. AI Annotation (LLM)

Models: Six leading LLMs were employed, split into "Lighter-weight" (GPT-4o-mini, Gemini-2.0-flash, Llama-3.1-8B) and "Full" (GPT-4o, Gemini-2.5-pro, Llama-3.3-70B).
Approach: Zero-shot learning with optimized prompts and structured output (True/False per category). Temperature was set to 0 for reproducibility.
Strategy: A "Wisdom of the Crowd" approach was applied across LLMs, using majority voting to aggregate predictions.

D. Human Annotation (Ground Truth)

Platform: Amazon Mechanical Turk (MTurk).
Scale: 34 unique annotators labeled a random subset of 1,000 posts (approx. 1% of the dataset).
Protocol: Each post received 5 annotations. Strict qualification criteria (Master status, high approval rate) were enforced.
Demographics: Annotator demographics (political ideology, affiliation, income, etc.) were collected to analyze bias.
Ground Truth: Derived via majority voting among the three human annotators per post.

E. Evaluation Metrics

Inter-Rater Reliability (IRR): Measured using Cohen's Kappa ( $\kappa$ ), Percentage Agreement, and Krippendorff's Alpha ( $\alpha$ ).
Performance: Precision, Recall, and F1-score were calculated by comparing LLM outputs against the human consensus labels.

3. Key Contributions

USE24-XD Dataset: The release of a publicly available, large-scale (nearly 100k posts), multi-label dataset of U.S. election-related content, enriched with metadata and curated annotations.
LLM Benchmarking: A comprehensive evaluation of six state-of-the-art LLMs across five nuanced harmful content categories, benchmarking them against human annotators.
Wisdom-of-the-Crowd LLM Strategy: Demonstration that aggregating predictions from diverse LLMs via majority voting yields higher consistency and reliability than single models, often surpassing human inter-annotator agreement.
Demographic Bias Analysis: Empirical evidence showing how human annotator demographics (specifically political ideology and affiliation) systematically shape labeling behavior, particularly for "Speculation" and "Hate Speech."

4. Key Results

A. LLM vs. Human Agreement

Consistency: LLMs exhibited higher internal consistency than humans. The mean Krippendorff's $\alpha$ for optimized LLM combinations was 0.70, compared to 0.62 for the best human annotation sets.
Category Performance:
- High Agreement: LLMs showed strong agreement on Conspiracy ( $\kappa \approx 0.75$ ) and Hate Speech.
- Low Agreement: Satire and Speculation showed lower agreement among both humans and LLMs due to ambiguity, though LLMs still outperformed humans in consistency.
Recall: When using human consensus as ground truth, LLM ensembles achieved high recall, reaching up to 0.90 for Speculation and 0.81 for Conspiracy.

B. Human Annotator Demographics

Ideological Influence: Political ideology significantly influenced labeling.
- Centrists/Independents were most likely to label content as Speculation.
- Conservatives/Republicans were more likely to label content as Hate Speech and less likely to label it as Speculation.
- Satire remained rare and showed low agreement across all groups, suggesting its ambiguity is inherent rather than demographic.
Implication: "Ground truth" in subjective tasks is not demographically neutral; annotator worldviews shape the dataset.

C. Cost-Benefit Analysis

Efficiency: LLM annotation is significantly cheaper and faster.
- Human Cost: ~$130 for 1,000 posts (at $15/hr).
- LLM Cost: Ranged from $0.05 (Gemini Flash) to $8.97 (Gemini 2.5 Pro) for 1,000 posts.
- Time: Humans took ~7.78 hours total; LLMs took seconds to minutes.
Conclusion: LLMs, particularly when aggregated, offer a scalable, cost-effective alternative to manual annotation with comparable or superior consistency.

5. Significance

Scalability for Research: The study provides a viable pathway for analyzing billions of social media posts daily, a task impossible with manual annotation alone.
Refining Harmful Content Detection: By moving beyond simple "misinformation" to include satire, speculation, and sensationalism, the framework captures a broader spectrum of democratic harm.
Methodological Shift: The paper validates the "Wisdom of the LLM Crowd" approach, suggesting that ensembles of diverse models can serve as robust, low-cost annotators for complex social science tasks.
Bias Awareness: The findings highlight that even in automated pipelines, human factors (demographics) influence the definition of "harm," urging future research to account for subjectivity in both human and AI labeling.

The USE24-XD dataset is publicly available to support future research in social media analysis, misinformation detection, and election integrity.