Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators

1. Problem Statement

Event annotation is critical for sociological research, particularly in monitoring trends like terrorism (e.g., the Global Terrorism Database, GTD). However, current workflows face significant bottlenecks:

• Cost and Efficiency: Expert human annotators are expensive and slow. While automated systems (like GDELT) scale well, they often suffer from low precision, high false positives, and lack of semantic understanding.
• Workflow Fragmentation: Existing NLP research often evaluates models on isolated subtasks (e.g., just event extraction) rather than holistic, end-to-end workflows that include document retrieval, cross-document coreference resolution (grouping documents about the same event), and variable extraction.
• Data Mismatch: Standard NLP datasets (e.g., ACE, ECB) are small, clean, and document-level, failing to reflect the noisy, multi-document, first-hand source reality of social science datasets like GTD.
• The Gap: There is a lack of empirical evidence on whether Large Language Models (LLMs) can replace experts or effectively assist them in realistic, complex annotation pipelines.

2. Methodology

The authors evaluated a holistic workflow using the Global Terrorism Database (GTD) as a case study. The workflow consists of two primary stages:

Stage A: Event Set Curation (Cross-Document Coreference Resolution)

The goal is to group noisy, duplicate, and conflicting documents into unique "event sets."

• Baseline: Traditional TF-IDF and nearest-neighbor search.
• LLM Approaches Tested:
  1. LLM-CLS (Pairwise Classification): Uses GPT-4o-mini to classify if two documents describe the same event. Includes a SEG (Segmentation) step where the LLM breaks long news digests into single-event blocks before classification.
  2. EMBEDDING: Groups documents based on semantic similarity (cosine similarity of vector embeddings) rather than keyword matching.
  3. K-LLMMEANS: An iterative clustering method using K-Means with LLM-generated summaries as cluster centroids.
• Evaluation: Precision, Recall, and F1 scores against a gold standard created by human experts.

Stage B: Variable Coding (Argument Extraction)

Once event sets are established, experts extract specific variables (e.g., target, perpetrator, casualties) based on a formal codebook.

• Experimental Conditions:
  1. Manual: Experts code variables without assistance.
  2. LLM-only: LLMs code variables independently.
  3. Hybrid (Human-in-the-Loop): Experts review and edit LLM-generated variable suggestions.
• Metrics:
  • Time: Total annotation time per event.
  • Adoption Rate: How often humans accept LLM suggestions.
  • Accuracy: Agreement rates measured via Normalized Match (NM), BERT Match (BEM), and PEDANTS (TF-IDF based F1).

3. Key Contributions

1. Holistic Workflow Evaluation: The paper moves beyond single-task evaluation to test LLMs in a realistic, multi-stage pipeline involving document curation and variable extraction.
2. Empirical Evidence on LLM Utility: It provides data showing that while LLMs cannot fully replace human experts, they serve as highly effective assistants.
3. Novel Clustering Method (K-LLMMEANS): The authors propose a cost-effective clustering method that combines high-quality embeddings with LLM summaries, outperforming pairwise classification in recall and cost-efficiency.
4. Error Analysis & Mitigation: The study identifies specific failure modes (e.g., hallucination when data is missing, ambiguity in "digest" documents) and proposes mitigation strategies like refined prompting and context injection.

4. Key Results

Event Set Curation

• Performance: The LLM-CLS+SEG method achieved the highest recall (0.66) and F1 (0.63), significantly outperforming the TF-IDF baseline (F1 0.10).
• Segmentation is Critical: Pre-processing long documents into single-event segments via LLMs drastically improved performance by reducing ambiguity.
• Cost vs. Performance: K-LLMMEANS offered a superior trade-off, achieving better recall and F1 than LLM-CLS at less than one-third of the computational cost.
• Conclusion: LLMs are excellent at finding similar documents (high recall) but still fall short of expert-level precision in defining unique event sets.

Variable Coding (Argument Extraction)

• Time Reduction: Providing LLM-generated variable suggestions to experts reduced annotation time by 25%.
• Adoption Rate: Experts adopted LLM suggestions 60% of the time overall. Adoption was highest for specific variables like "Country" (92%) and lowest for vague variables like "Location" (40%).
• Human-LLM Agreement: In hybrid settings, human-LLM agreement was nearly identical to human-human agreement (approx. 55-60%), suggesting LLMs provide "near-human" utility as a first draft.
• Model Size: Larger models did not significantly outperform smaller ones (e.g., GPT-4o-mini vs. Mistral 8x7B).
• Hallucination: Models struggled significantly with "Not Available" (NA) scenarios. When information was missing, models often hallucinated values (Recall for NA was ~0.25), with Mixtral-8x7B showing particularly poor recall (0.03) despite high precision.

5. Significance and Conclusion

The paper concludes that LLMs are effective human annotation assistants but not good independent annotators for complex social science tasks.

• Practical Impact: Integrating LLMs into the workflow allows organizations to scale data collection without sacrificing quality, as the "human-in-the-loop" model leverages the speed of AI and the judgment of experts.
• Strategic Shift: The findings suggest a shift from trying to automate the entire pipeline to using LLMs for augmentation. This reduces the cognitive load on annotators, allowing them to focus on verification and complex reasoning rather than initial data entry.
• Future Directions: The authors recommend fine-tuning models on domain-specific codebooks, using task-reward mechanisms, and employing prompt engineering to mitigate hallucinations in missing data scenarios.

In summary, the study validates that while LLMs cannot yet replace the nuanced judgment required for event annotation, they are powerful tools that can significantly accelerate the process when used as a collaborative aid.