Technical Summary: Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

Problem Statement

Fine-grained opinion analysis, specifically tasks like Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) extraction, requires detailed identification of sentiment polarity, target entities (aspect terms), opinion spans, and implicit categories. While highly valuable for understanding nuanced sentiments, creating datasets for these tasks demands significant human effort and strict adherence to complex guidelines. Existing datasets are often small, domain-specific (e.g., SemEval laptop and restaurant reviews), and insufficient for diverse real-world applications. The authors investigate whether Large Language Models (LLMs) can serve as fully autonomous, cost-effective alternatives to human annotators for these complex, relational extraction tasks, or if they are better suited as assistive tools.

Methodology

The authors propose a declarative, inference-time adaptation pipeline that avoids task-specific fine-tuning. The methodology consists of three primary stages:

1. Declarative Annotation Pipeline:

   • Framework: The pipeline utilizes DSPy to "program" LLMs using structured, declarative modules rather than manual prompt engineering. This reduces variability and spurious interactions between prompt selection and model performance.
   • Optimization: The MIPROv2 prompt optimizer is employed to systematically refine instructions and in-context learning (ICL) examples. The pipeline is optimized on a development split (D1) using few-shot levels ($k \in \{5, 10, 15\}$).
   • Tasks: The system targets two formulations:
     • ASTE: Extracting triplets $\{aspect\_term, sentiment, opinion\_span\}$.
     • ACOS: Extracting quadruples $\{aspect\_term, sentiment, opinion\_span, aspect\_category\}$, which includes handling implicit aspects and categories.

2. Redundant Generation:

   • Multiple LLMs of varying sizes (Mini $\approx$4B, Small $\approx$14B, Medium $\approx$32B) are deployed as independent annotators ($A_1, A_2, A_3$) on a held-out development split (D2).
   • This generates multiple candidate annotation sets for each input text, creating a dataset of redundant predictions.

3. LLM-Based Adjudication:

   • A dedicated adjudication pipeline, powered by the best-performing annotator model (A1), aggregates the multiple candidate annotations.
   • Unlike traditional majority voting (which is impractical for open-ended, combinatorial triplet/quadruple outputs), the adjudicator uses an LLM to resolve inter-annotator disagreements and synthesize a final output. This mimics the human process of arbitration.

Key Contributions

• Declarative Annotation Pipeline: A schema-constrained pipeline using DSPy that enables systematic interaction for extracting opinions without hand-crafted prompts.
• LLM as Adjudicator: A novel methodology for using an LLM to aggregate and resolve conflicts between multiple LLM-driven annotations, specifically designed for the non-discrete output space of ASTE and ACOS.
• Scale Assessment: An empirical evaluation of the trade-off between model parameter size and annotation efficiency across Mini, Small, and Medium scales.
• Reliability Analysis: A granular error analysis using element-wise and pairwise metrics, as well as Krippendorff's $\alpha$ for inter-annotator agreement, revealing the specific strengths and weaknesses of LLMs in this domain.

Results and Findings

The evaluation, conducted on standard SemEval-derived datasets (Laptop and Restaurant domains), reveals a critical performance bifurcation:

1. Span-Level Reliability vs. Structural Complexity:

   • LLMs demonstrate high fidelity in identifying individual spans (aspect terms, opinion spans, and sentiment polarity). For instance, sentiment polarity alignment often exceeds 90% F1.
   • However, performance degrades significantly as relational complexity increases. The ability to correctly link spans to their targets (e.g., matching an opinion to the correct aspect) and handle implicit elements (especially in ACOS) is a major bottleneck.
   • In the ACOS task, which involves a high-entropy search space (112 categories in the Laptop domain), LLMs struggle significantly compared to supervised fine-tuning (SFT) baselines.

2. Impact of Model Scale and Reasoning:

   • ASTE: In restaurant domains, inference-time adaptation methods often outperform SFT baselines, suggesting LLMs leverage emergent reasoning for generalization. In the laptop domain, SFT remains superior.
   • Adjudication: Adjudication generally improves F1 scores for Mini and Medium models. However, for Small models (specifically Qwen3-14B), adjudication sometimes reduced performance. The authors attribute this to the lack of "long-chain reasoning states" (thinking capabilities) in certain distilled models, which are crucial for resolving complex relational conflicts.
   • ACOS: While adjudication improves results, the absolute F1 scores remain low (e.g., ~18.99% for Laptop, ~40.53% for Restaurant in the Medium tier), highlighting the difficulty of implicit aspect/category detection without domain-specific training.

3. Inter-Annotator Agreement (IAA):

   • Krippendorff's $\alpha$ scores indicate that reliability increases with model size. Triplet elements approach high reliability, while quadruple elements (ACOS) remain moderately reliable at best.

Significance and Conclusion

The paper concludes that while LLMs are not yet ready to fully replace human annotators for fine-grained opinion analysis due to their struggles with relational structures and implicit elements, they possess strong utility as high-fidelity annotation assistants and data augmentation tools.

The authors argue that the current state of LLMs allows them to reliably extract individual spans and sentiment polarities. Therefore, the most effective application of these models is to expand existing fine-grained opinion-annotated datasets or assist human annotators, rather than attempting to generate ground-truth annotations autonomously for complex, relational tasks. The work underscores the importance of adjudication mechanisms and the specific need for reasoning capabilities (beyond raw parameter scaling) in handling the structural complexities of opinion mining.