Label-Consistent Data Generation for Aspect-Based Sentiment Analysis Using LLM Agents

1. Problem Statement

Aspect-Based Sentiment Analysis (ABSA) involves identifying specific aspect terms in text and determining the sentiment polarity associated with each. Despite its utility in applications like customer feedback analysis, ABSA faces significant challenges:

• Data Scarcity: Fine-grained, aspect-level annotations are expensive and time-consuming to produce, leading to limited dataset coverage.
• Model Limitations: Models struggle with rare aspect-sentiment combinations and domain-specific linguistic variations.
• Synthetic Data Issues: While Large Language Models (LLMs) have been used to generate synthetic training data, existing approaches rely heavily on single-step prompting. This often results in:
  • Label Inconsistency: The generated text does not match the intended aspect or sentiment labels.
  • Structural Deficiencies: Lack of diversity and failure to handle complex edge cases.
  • Noise: High rates of errors that degrade downstream model performance when used for training.

The authors investigate whether an agentic, multi-step generation workflow can overcome these limitations by separating generation from verification, thereby producing high-quality, label-consistent synthetic data.

2. Methodology

The paper proposes a ReAct-style (Reasoning + Acting) agentic framework using the Qwen2.5-14B model. The system employs two cooperating agents with distinct roles, utilizing specific tools to ensure data quality.

A. Agentic Workflow

1. Generator Agent:
   • Policy Extraction: Samples aspect terms and sentiment polarities from real SemEval training data. It also extracts writing style, grammatical structure, and sentence length from a small set of real reviews to guide stylistic variety.
   • Sentence Generation: Uses the extracted policy to generate candidate sentences that strictly adhere to the sampled aspect-polarity pairs and stylistic constraints.
   • Output: Produces the sentence along with machine-readable Terms= and Polarity= annotations.
2. Evaluator Agent:
   • Label Inclusion Check: Verifies that the generated sentence contains all required aspect terms exactly as specified (checking for pluralization errors, missing terms, or partial matches).
   • Label Verification: Prompts the LLM to judge if the sentiment polarity in the sentence aligns with the intended label.
   • Filtering: Only examples passing both structural and semantic checks are saved. Failed examples trigger a regeneration loop.

B. Baseline Comparison

To isolate the effect of the agentic structure, the authors developed a Prompt-Based Baseline. This baseline uses the exact same model (Qwen2.5-14B), the same aspect-polarity sampling strategy, and nearly identical prompts, but generates data in a single monolithic step without style extraction, tool usage, or verification.

C. Experimental Setup

• Datasets: Four SemEval datasets (Laptop14, Rest14, Rest15, Rest16).
• Subtasks: Aspect Term Extraction (ATE), Aspect Sentiment Classification (ATSC), and Aspect Sentiment Pair Extraction (ASPE).
• Models: Fine-tuned T5-Base (standard encoder-decoder) and Tk-Instruct-base (heavily instruction-tuned on 1,600+ tasks).
• Training Configurations:
  1. Original data only.
  2. Synthetic data only.
  3. Mixed data (Original + Synthetic) at ratios of 1:1 (x1) and 1:2 (x2).

3. Key Contributions

1. Agentic Workflow for ABSA: Introduced a multi-step pipeline that decomposes data generation into style extraction, controlled generation, and automatic verification, ensuring strict adherence to complex ABSA constraints.
2. Isolated Baseline: Designed a rigorously matched prompting baseline to prove that performance gains stem from the agentic structure (reasoning/verification) rather than the underlying model or prompts alone.
3. Systematic Analysis: Provided a comprehensive evaluation of how model pretraining levels (T5 vs. Tk-Instruct), task complexity (ATE/ATSC/ASPE), and augmentation scales impact the efficacy of synthetic data.

4. Key Results

A. Label Consistency

• Agentic vs. Prompting: The agentic approach significantly outperformed raw prompting in label preservation.
  • ATE Consistency: Agentic (78.17%) vs. Prompting (43.89%).
  • ASPE Consistency: Agentic (33.89%) vs. Prompting (18.33%).
• The verification steps effectively filtered out structural deviations and polarity mismatches common in naive prompting.

B. Downstream Performance (F1 Scores)

• Synthetic Data Only: Training exclusively on synthetic data resulted in a substantial performance drop for both models, confirming that synthetic data cannot yet replace human annotations.
• Mixed Data (Original + Synthetic):
  • T5-Base: Showed consistent gains across all subtasks when mixed with agentic data (e.g., +1.93 F1 on Rest16 for ATE). The agentic augmentation allowed T5-Base to approach or even surpass the performance of the heavily pretrained Tk-Instruct.
  • Tk-Instruct: Showed marginal or inconsistent gains. Its strong instruction-tuning prior made it less sensitive to additional synthetic variation, and in some cases (ASPE), performance slightly degraded.
  • Prompting Baseline: Often failed to improve performance and sometimes degraded it, particularly in complex tasks like ASPE.

C. Task Sensitivity

• ATSC (Classification): Benefited most from augmentation as it requires only sentiment assignment to a known aspect.
• ATE (Extraction): Showed moderate gains; challenges arose from ambiguity in generated aspect spans.
• ASPE (Pair Extraction): Showed the least improvement. The compound difficulty of extracting spans and assigning sentiment made the task highly sensitive to even minor label inconsistencies in synthetic data.

D. Scaling Effects

• Increasing the synthetic data ratio from 1:1 (x1) to 1:2 (x2) generally did not yield additional gains and often reduced performance. This suggests that larger quantities of synthetic data amplify noise rather than providing useful signal.

5. Significance and Conclusion

• Bridging the Pretraining Gap: The study demonstrates that task-specific agentic augmentation can partially substitute for massive-scale instruction tuning. A smaller model (T5-Base) augmented with high-quality synthetic data can match the performance of a much larger, instruction-tuned model (Tk-Instruct).
• Quality over Quantity: The results emphasize that controlled generation and verification are critical. Naive prompting, even with the same model, introduces too much noise to be effective for structured tasks like ABSA.
• Practical Scalability: The agentic framework offers a scalable, automated method to enhance ABSA models without requiring extensive human annotation or expensive proprietary LLMs, provided the generation is strictly verified.

Future Directions: The authors suggest extending this framework to multilingual settings, decomposing the pipeline into specialized generator/validator models, and incorporating "model-in-the-loop" feedback to target specific model weaknesses.