Are LLMs Ready to Replace Bangla Annotators?

1. Problem Statement

The paper addresses the growing reliance on Large Language Models (LLMs) as automated annotators to scale dataset creation, particularly for low-resource languages. While this approach offers efficiency, there is a critical lack of understanding regarding the reliability and bias of LLMs in sensitive contexts. Specifically, the authors focus on Bangla hate speech detection, a task characterized by:

• Low-resource constraints: Limited high-quality annotated data compared to English.
• Identity sensitivity: The potential for severe downstream consequences if annotators misclassify content related to specific identities.
• Inherent ambiguity: Even human annotators struggle to reach consensus on hate speech definitions, making the benchmarking of automated systems particularly challenging.

The core problem is determining whether current LLMs can serve as unbiased, stable replacements for human annotators in these complex, high-stakes scenarios.

2. Methodology

The authors employed a systematic benchmarking approach to evaluate the performance of LLMs as zero-shot annotators. Key methodological components include:

• Model Scope: A comprehensive evaluation of 17 different LLMs, covering a range of sizes and architectures.
• Task: Zero-shot classification of Bangla hate speech (no fine-tuning on the specific dataset was performed).
• Evaluation Framework: A unified framework designed to measure not just accuracy, but also annotator bias and judgment stability.
• Comparative Analysis: The study specifically investigates the correlation between model scale (parameter count) and annotation quality, testing the hypothesis that larger models inherently perform better.

3. Key Contributions

• Systematic Benchmarking: The paper provides one of the first rigorous evaluations of multiple LLMs acting as zero-shot annotators specifically for the Bangla language in the context of hate speech.
• Bias and Instability Detection: It introduces a framework to quantify annotator bias and instability (inconsistency in judgments) in LLMs, moving beyond simple accuracy metrics.
• Counter-Intuitive Findings on Scale: The work challenges the prevailing assumption that "bigger is better," demonstrating that model scale does not linearly correlate with annotation reliability in low-resource, sensitive tasks.

4. Key Results

• Substantial Instability: The analysis revealed significant instability in model judgments, indicating that LLMs often produce inconsistent labels for similar inputs.
• Presence of Bias: The models exhibited clear annotator biases, which could lead to unfair or harmful downstream outcomes in identity-sensitive contexts.
• Scale vs. Performance Paradox: Contrary to expectations, increased model scale did not guarantee improved annotation quality.
  • Smaller, Task-Aligned Models: Surprisingly, smaller models that were more specifically aligned with the task demonstrated more consistent behavior than their larger, general-purpose counterparts.
• Human Agreement Context: The difficulty of the task was underscored by the fact that even human agreement is low, yet LLMs struggled to match or exceed the reliability of human consensus.

5. Significance and Implications

• Caution Against Blind Deployment: The findings serve as a critical warning against the uncritical replacement of human annotators with LLMs for sensitive tasks in low-resource languages.
• Need for Rigorous Evaluation: The paper underscores the necessity of evaluating LLMs for bias and stability before deployment, rather than relying solely on scale or general performance metrics.
• Strategic Model Selection: For practitioners, the results suggest that in specific, sensitive annotation tasks, smaller, specialized models may be a more reliable choice than massive foundation models.
• Future Research Direction: The work highlights the gap in current LLM capabilities regarding low-resource, identity-sensitive NLP tasks, calling for further research into bias mitigation and robust evaluation frameworks for these languages.