DanceHA: A Multi-Agent Framework for Document-Level Aspect-Based Sentiment Analysis

This paper introduces DanceHA, a multi-agent framework that utilizes a divide-and-conquer strategy and human-AI collaboration to address the underexplored challenge of document-level Aspect-Based Sentiment Intensity Analysis (ABSIA) in informal writing styles, accompanied by the release of the new Inf-ABSIA dataset.

The Gist

Imagine you are trying to understand the mood of a massive, chaotic party by reading thousands of text messages sent by the guests. Some guests are polite and formal ("The food was good"), while others are wild, informal, and expressive ("The food was soooooo goooood!!!!").

This paper introduces a new system called DanceHA designed to do exactly that: read long, messy reviews (like those on Yelp or Amazon) and figure out exactly what people are talking about, how they feel about it, and how strongly they feel.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Wall of Text"

Traditional AI is like a student trying to read a 50-page novel in one sitting. It often gets tired, misses details, or confuses one character with another.

• The Challenge: Most AI is trained on short sentences. But real reviews are long documents.
• The "Informal" Twist: People don't just say "good." They say "goooood," "amazzzing," or use emojis. Standard AI often ignores these "lengthened" words, missing the fact that the person is extremely happy, not just "okay."

2. The Solution: The "Dance" (Divide and Conquer)

Instead of one giant brain trying to do everything, the authors created a team of specialized robots (agents) that work together. They call this Dance.

Think of it like a construction crew building a house:

• The Divider (The Foreman): Instead of one person trying to build the whole house, the Foreman breaks the long review into small, manageable chunks. If a review talks about the "battery," the "screen," and the "keyboard," the Foreman separates these into three different work zones.
• The Specialists (The Workers):
  • The Category Expert: Looks at the "battery" zone and says, "This belongs in the 'Hardware' category."
  • The Opinion Hunter: Looks at the text and finds the specific words. If someone wrote "baaaattery," this agent catches that specific spelling and keeps it (because the extra 'a's matter!).
  • The Mood Reader: Decides if the feeling is positive or negative and, crucially, how intense it is. Is it a "meh" (score 1) or "OMG I LOVE IT" (score 5)?
• The Merger: Once the specialists finish their small jobs, their results are stitched back together into a neat list.

3. The "HA" (Human-AI Collaboration)

Even the best robot team makes mistakes. So, the authors added a second layer called HA (Human-AI).

• Imagine the robot team produces a draft of the review analysis.
• A Manager Robot looks at the drafts from three different robot teams and tries to agree on the best answer.
• Then, Human Reviewers step in. They act like the final editors. They check the robot's work, fix errors, throw out bad guesses, and add anything the robots missed.
• The Result: This process created a brand new, super-accurate dataset called Inf-ABSIA, which is like a "Gold Standard" textbook for teaching AI how to read informal reviews.

4. The "Secret Sauce": Teaching the Small Models

The paper also shows how to teach smaller, cheaper AI models to be as smart as the big, expensive ones.

• They took the "reasoning steps" (the thought process) the big robot team used to solve the problems.
• They fed these steps to a smaller student model (like a Qwen-14B).
• The Analogy: It's like a master chef (the big AI) writing down their secret recipe and cooking techniques, then teaching a junior chef (the small AI). The junior chef ends up cooking almost as well as the master, but much faster and cheaper.

Why Does This Matter?

• It catches the "vibe": It understands that "cooool" is stronger than "cool."
• It handles long stories: It doesn't get lost in long paragraphs.
• It's accurate: By using a team of robots + humans, it creates data that is much better than what we had before.

In a nutshell: DanceHA is a smart, team-based system that breaks down long, messy, informal reviews into tiny pieces, assigns experts to solve each piece, checks the work with humans, and then teaches smaller AIs how to do the same thing. It's the difference between trying to eat a whole pizza in one bite versus slicing it up and enjoying every single slice.

Technical Summary

1. Problem Definition

The paper addresses Document-Level Aspect-Based Sentiment Intensity Analysis (ABSIA), a complex task that remains underexplored compared to sentence-level ABSA. The specific challenges include:

• Complexity: Extracting ACOSI tuples (Aspect, Category, Opinion, Sentiment Polarity, Sentiment Intensity) from long, unstructured documents rather than single sentences.
• Informal Styles: User-generated content (UGC) often contains informal linguistic features like word lengthening (e.g., "coool," "goooood!!!!") and excessive punctuation, which intensify sentiment but are often ignored by standard models.
• Data Scarcity: There is a lack of large-scale, fine-grained, labeled datasets for document-level ABSIA, making supervised training difficult.
• Context Length: Processing long documents requires handling context that exceeds the efficient processing limits of single-pass LLM inference without losing detail.

2. Methodology: The DanceHA Framework

The authors propose DanceHA, a multi-agent framework designed for open-ended document-level ABSIA. It consists of two main components: Dance (Divide-and-Conquer Teamwork) and HA (Human-AI collaboration).

A. Dance: Divide-and-Conquer Teamwork

This component decomposes the complex document-level task into manageable sub-tasks using specialized LLM agents. It operates in three stages:

1. Divider (Aspect-Based Thought Grouping):
   • An agent ($F_D$) splits the long document into "thought groups."
   • It identifies explicit aspect terms and groups consecutive sentences pertaining to the same aspect into a single semantic unit.
   • Goal: Reduce context complexity and isolate specific aspects for focused analysis.
2. Conquer (Specialized Agent Collaboration):
   Three specialized agents process the thought groups in parallel or sequence:
   • Category Assignment Agent ($F_C$): Uses Retrieval-Augmented Generation (RAG) with a domain-specific category list to assign the correct category (e.g., "food quality") to the aspect.
   • Opinion Extraction Agent ($F_O$): Extracts opinion words/phrases, explicitly preserving informal stylistic features (lengthening, punctuation) from the original text.
   • Sentiment & Intensity Analysis Agent ($F_S$): Determines sentiment polarity (Positive/Negative) and assigns a Sentiment Intensity Score (SIS) on a 0–5 Likert scale.
3. Merge:
   • A rule-based integration method combines outputs from the agents to form a structured list of ACOSI tuples.

B. HA: Human-AI Collaboration for Annotation

To address the lack of labeled data, the authors introduce a pipeline for dataset creation:

• Manager Agent (MA): Multiple "Dance teams" (using different backbone LLMs) generate candidate annotations for the same document. A Manager Agent aggregates these outputs, resolving conflicts and creating a consensus-driven preliminary label set.
• Human Revision: Human annotators review the Manager Agent's output, retaining high-quality labels, correcting inconsistencies, discarding low-confidence predictions, and adding missing labels.
• Result: This process creates the Inf-ABSIA dataset.

C. Knowledge Distillation

The authors construct Reasoning Chains from the multi-agent workflow (Aspect Grouping $\to$ Category $\to$ Opinion $\to$ Sentiment/Intensity $\to$ Merge). These chains are used to fine-tune smaller "student" models (e.g., Qwen-14B) via Supervised Fine-Tuning (SFT) with LoRA, transferring the multi-agent reasoning capabilities to a single model.

3. Key Contributions

1. DanceHA Framework: A novel, training-free multi-agent framework that effectively handles document-level ABSIA with informal styles using a divide-and-conquer strategy.
2. Inf-ABSIA Dataset: A new, high-quality dataset containing 2,714 long-context documents across three domains (Laptop, Restaurant, Hotel) with 23,024 fine-grained ACOSI tuples. It is the largest document-level ABSIA dataset to date.
3. Insight on Informal Styles: Empirical evidence showing that informal expressions (e.g., lengthened words) significantly correlate with higher sentiment intensity scores, a nuance often missed by standard models.
4. Effective Knowledge Distillation: Demonstrating that reasoning chains generated by the multi-agent framework can be distilled into smaller models, allowing a fine-tuned Qwen-14B to outperform few-shot GPT-4o and match the performance of a much larger Qwen2.5-72B.

4. Experimental Results

The framework was evaluated on three domains (Restaurant, Hotel, Laptop) using 7 different LLM backbones (including GPT-4o, DeepSeek-V3, and Qwen series).

• Performance vs. Baselines:
  • Dance significantly outperforms Zero-shot and Few-shot Chain-of-Thought (CoT) baselines.
  • Manager Agent (MA) further boosts performance. For example, in the Restaurant domain, MA with DeepSeek-V3 achieved an F1 score of 63.18%, substantially beating the best single-agent Dance setup (47.80% with GPT-4o).
• Ablation Studies:
  • Removing the Divide-and-Conquer (D&C) strategy caused a significant performance drop (average F1 drop of ~8.4%), proving the necessity of decomposing long documents.
  • Removing Teamwork (using a single agent for all sub-tasks) also reduced performance, highlighting the benefit of specialized agents.
• Distillation Results:
  • A Qwen-14B model fine-tuned with reasoning chains achieved an F1 of 34.52% (Restaurant), outperforming Few-shot CoT GPT-4o (35.04% is close, but the distilled model is smaller and more efficient) and matching the performance of the much larger Qwen2.5-72B.
• Human Evaluation:
  • The Inf-ABSIA dataset labels received an average human rating of 4.47/5 with an Inter-Annotator Agreement (IAA) of 0.61, confirming high data quality.

5. Significance

• Advancing Document-Level Analysis: The work bridges the gap between sentence-level and document-level sentiment analysis, tackling the specific challenges of long-context reasoning and multi-aspect extraction.
• Handling Informal Language: It provides a robust methodology for processing the "messy" reality of user reviews (emojis, lengthening), showing that these features are critical for accurate intensity estimation.
• Scalable Annotation: The DanceHA pipeline offers a viable solution for creating high-quality, fine-grained datasets in domains where manual labeling is too costly, leveraging multi-agent consensus and human-in-the-loop verification.
• Efficiency via Distillation: It demonstrates that complex multi-agent reasoning can be compressed into smaller, efficient models, making advanced ABSIA accessible without requiring massive compute resources for inference.