💬 NLP

Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection

While instruction-tuned LLMs can generate large-scale annotations for hostility detection at a fraction of the cost of human labeling with comparable aggregate performance, they introduce systematic error patterns in ambiguous cases that necessitate careful consideration of application-specific error profiles over simple cost-efficiency metrics.

Original authors: Ahmad Dawar Hakimi, Lea Hirlimann, Isabelle Augenstein, Hinrich Schütze

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmad Dawar Hakimi, Lea Hirlimann, Isabelle Augenstein, Hinrich Schütze

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to spot "toxic" comments on a social media platform, specifically comments that are hostile toward immigrants. You have a massive library of 278,000 comments, but you can't read them all yourself. You need to label them first so the robot can learn.

This paper asks a very modern question: Do we still need humans to do this labeling, or can we just hire a super-fast, cheap AI to do it for us?

Here is the breakdown of their experiment, explained with simple analogies.

The Setup: The "Taste Test"

The researchers had two ways to label the comments:

The Human Panel: They hired 6 real people to read the comments and decide if they were hostile. This is expensive and slow (like hiring a team of food critics).
The AI Panel: They used a powerful AI (GPT-5.2) to read the same comments. This is incredibly fast and cheap (like using a high-tech flavor scanner).

They also tested Active Learning. Think of this as a "smart sampling" strategy. Instead of reading 100% of the comments, the AI tries to pick out the most confusing or interesting ones for the humans to read, hoping to learn the rules faster with fewer examples.

The Big Discovery: The "Volume vs. Quality" Trade-off

1. The Cost Miracle
The researchers found that if you let the AI label 26,000 comments (a huge amount), the resulting robot classifier was just as good as the one trained on 3,800 human labels.

The Catch: The AI version cost $43. The human version cost $316.
The Analogy: It's like buying a massive bag of generic brand cereal for $5 that tastes just as good as a small box of premium artisanal cereal for $35. If you just want to fill your bowl (get a good score), the cheap, massive bag wins.

2. The "Active Learning" Trap
They tested if "Active Learning" (the smart sampling) helped humans work faster.

The Result: It didn't really help. In this specific case, the pool of comments was already "pre-filtered" to be about immigration. Because the pool was already full of relevant stuff, picking the "most confusing" ones didn't save much time or money compared to just picking random ones.
The Analogy: Imagine you are looking for red marbles in a bucket that is already 90% full of red marbles. Trying to be "smart" about which ones you pick doesn't help; you're just wasting time. You might as well just grab a handful randomly.

The Twist: The "Over-Enthusiastic" AI

Here is where it gets interesting. While the AI-trained robot got the same overall score as the human-trained robot, they made different kinds of mistakes.

The Human Robot: Was balanced. It made mistakes where it missed a toxic comment, and mistakes where it wrongly flagged a harmless one.
The AI Robot: Was over-enthusiastic. It flagged way more comments as "toxic" than the humans did.

The "Border Control" Analogy:
Imagine a comment says: "Who opened the borders? We need controls!"

The Human thinks: "This is a critique of a government policy. It's political, but it's not attacking a specific group of people. Safe."
The AI thinks: "This mentions borders and control. It sounds negative toward immigrants. Toxic!"

The AI tends to confuse criticizing a policy with hating a group of people. Because the AI labeled so many things as "toxic," the robot it trained became a "guilty until proven innocent" machine. It catches almost all the bad stuff, but it also catches a lot of innocent stuff.

The "Confidence" Problem

The AI-trained robot was also overconfident in its mistakes.

When a human-trained robot made a mistake, it was usually unsure (low confidence).
When the AI-trained robot made a mistake, it was 100% sure it was right.

The Analogy:

Human Robot: "I think this comment might be bad, but I'm not 100% sure. Let's ask a human to double-check."
AI Robot: "This comment is definitely bad! I am 100% sure! No need to check!"
The Risk: If you rely on the AI robot to filter content automatically, it might delete innocent comments without anyone noticing because it was so confident it was right.

The Conclusion: Do We Still Need Humans?

Yes, but it depends on your goal.

If you are a researcher with a tiny budget who just needs a general model to see "what's happening" in the data: Use the AI. It's 7 times cheaper and almost as accurate.
If you are a content moderator where false alarms (deleting innocent posts) are a disaster: Stick with Humans (or a hybrid). The AI's tendency to over-flag subtle, ambiguous comments is dangerous.

The Final Metaphor:
Think of the AI annotator as a security guard who is terrified of missing a threat. He stops everyone who looks even slightly suspicious. He catches all the bad guys, but he also stops a lot of innocent tourists.
The Human annotator is a security guard who is good at reading context. He knows the difference between a tourist holding a map and a terrorist holding a weapon. He misses a few bad guys, but he doesn't stop the innocent tourists.

The paper's advice: Don't just look at the final score (the F1 score). Look at how the mistakes happen. If you can't afford to make "false positive" mistakes (accusing innocent people), you still need humans in the loop, even if the AI is cheaper.

1. Problem Statement

The paper addresses the bottleneck of data annotation in Natural Language Processing (NLP), specifically within the context of Active Learning (AL). While instruction-tuned Large Language Models (LLMs) can annotate vast corpora at negligible cost, it remains unclear:

Whether LLM labels can effectively replace human labels within an AL loop.
Whether AL remains necessary when entire corpora can be labeled instantly by LLMs.
How the source of annotation (Human vs. LLM) affects the error structure and behavior of downstream classifiers, particularly in nuanced tasks like detecting anti-immigrant hostility where the distinction between policy critique and group-directed hate is subtle.

2. Methodology

Dataset Construction

Source: 467,660 public comments from the official TikTok accounts of six major German political parties (CDU/CSU, Greens, SPD, FDP, AfD, Die Linke) collected between Jan 2024 and Sep 2025.
Preprocessing: A four-stage pipeline removed duplicates, non-informative content (emojis, URLs), and non-German text, resulting in 277,902 German comments.
LLM Prefiltering: A two-stage LLM pipeline was used to construct the annotation pool:
1. Topic Prefiltering: Llama-3.3-70B filtered for immigration-related content, reducing the pool to 25,974 comments.
2. Annotation: GPT-5.2 labeled these comments as ANTI-IMMIGRANT or NOT ANTI-IMMIGRANT using a single holistic prompt.
Human Annotation: A subset of 5,000 comments was independently annotated by six native/proficient German crowdworkers via Prolific using a two-question framework:
- Q1: Does it refer to immigrants/policy?
- Q2: Is the portrayal negative/hostile?
- Final Label: ANTI-IMMIGRANT only if Q1=Yes AND Q2=Yes.
- This yielded 1,200 gold evaluation items (multi-annotated) and 3,800 training items.

Experimental Design

The study compared seven experimental conditions across four transformer encoders (german-bert, ModernGBERT, gbert-base, xlm-r-base) and 10 random seeds:

Source: Human vs. LLM.
Sampling Strategy: Active Learning (Entropy/BALD) vs. Random Sampling vs. Full Pool.
Volume: Ranging from ~530 (AL limit) to 25,974 (Full LLM pool).

Key Conditions:

AL-HUMAN / AL-LLM: Active learning loops (max 530 labels).
FULL-HUMAN: 3,800 human labels.
FULL-LLM: 3,800 LLM labels.
FULL-LLM-26K: Full LLM pool (25,974 labels).

Evaluation Metrics

Primary: Macro-F1 and Class-specific F1 for the ANTI-IMMIGRANT class (F1-Anti).
Secondary: Area under the Learning Curve (ALC), cost-effectiveness analysis, and error profile analysis (False Positives vs. False Negatives).

3. Key Contributions

Large-Scale Empirical Comparison: A rigorous comparison of human vs. LLM annotation in AL settings using a novel dataset of 277,902 German political TikTok comments.
Cost-Performance Frontier: Demonstrates that scaling LLM annotation volume can compensate for lower label quality, achieving human-level performance at 1/7th the cost.
Error Structure Analysis: Reveals that while aggregate metrics (F1) may be comparable, LLM-trained classifiers exhibit a systematic False Positive (FP) bias, over-predicting hostility in ambiguous policy discussions.
AL Efficacy in Pre-Enriched Pools: Shows that in pools already filtered for relevance, Active Learning offers little advantage over random sampling compared to the cost of human annotation.

4. Key Results

Performance & Cost

Cost Efficiency:
- Full Human (3,800 labels): $316 ($ 0.083/label).
- Full LLM (3,800 labels): $6.34 ($ 0.002/label).
- Full LLM (25,974 labels): $43.30.
Performance:
- At matched volume (3,800 labels), FULL-HUMAN outperformed FULL-LLM by 0.016–0.035 F1-Macro points.
- However, FULL-LLM-26K (7x more data) achieved comparable F1-Macro to FULL-HUMAN and higher F1-Anti (better recall), effectively closing the quality gap through volume.
- Active Learning: AL with human labels provided a marginal F1 advantage over random sampling but was significantly more expensive than LLM annotation. AL with LLM labels performed worse than full LLM annotation at the same cost.

Error Profiles & Divergence

Systematic Bias: LLM-trained classifiers (especially FULL-LLM-26K) showed a massive asymmetry in errors: 243 False Positives vs. 21 False Negatives (vs. 124 FP / 87 FN for Human).
Confidence: LLM-trained models assigned higher confidence to their False Positives than human-trained models, making them harder to filter via confidence thresholds in deployment.
Source of Divergence: The discrepancy concentrates in topically ambiguous discussions (e.g., "Border Control," "Welfare State").
- Human Annotators: Strictly separated policy critique from group hostility (via the two-question framework).
- LLM: Treated negative framing of immigration-adjacent topics as sufficient evidence for ANTI-IMMIGRANT, conflating policy criticism with group-directed hate.

Active Learning Findings

In the pre-enriched pool (already filtered for immigration relevance), Entropy-based acquisition slightly outperformed BALD, but neither provided a significant advantage over random sampling.
This suggests that for pre-filtered datasets, the marginal benefit of uncertainty-based selection is negligible compared to the cost of human labeling.

5. Significance and Implications

Rethinking Active Learning: For tasks where LLMs can cheaply label the entire relevant corpus, the "cost premium" of human-in-the-loop Active Learning is often unjustified. Scaling LLM data is a more cost-effective strategy than optimizing human acquisition functions.
Annotation Strategy Matters: Aggregate F1 scores mask critical differences in error types. If the application requires high precision (minimizing false accusations of hate speech), human annotation (or LLMs with stricter, human-aligned prompts) is superior. If the goal is high recall (catching all potential hate), scaled LLM annotation is viable.
Ethical Considerations: LLM annotation eliminates the psychological burden on human annotators exposed to toxic content, a significant ethical advantage.
Guidance for Practitioners:
- Use LLM annotation at scale for budget-constrained projects requiring broad coverage.
- Use Human annotation when the definition of harm is culturally nuanced, requires strict separation of policy vs. people, or when false positives carry high real-world costs.
- Do not rely solely on F1: Analyze error profiles (FP/FN ratios) and confidence distributions to choose the right annotation source for the specific application.

Conclusion

The paper concludes that while LLMs can replace humans for aggregate performance at a fraction of the cost, they introduce a distinct error profile characterized by over-prediction of hostility in ambiguous contexts. The choice between human and LLM annotation should not be driven by cost or F1 alone, but by the acceptable error profile for the target application.