Extending Czech Aspect-Based Sentiment Analysis with Opinion Terms: Dataset and LLM Benchmarks

This paper introduces a novel Czech restaurant-domain dataset enriched with opinion terms for aspect-based sentiment analysis, benchmarks various Transformer and LLM models across multiple settings, and proposes an LLM-driven translation and label alignment methodology to effectively address cross-lingual challenges in low-resource languages.

The Gist

Imagine you are a restaurant critic, but instead of writing a long review, you are a robot trying to understand exactly what people like and dislike about a meal.

This paper is about teaching that robot to speak Czech and understand the tiny, specific details of a restaurant review, not just the general vibe.

Here is the story of the paper, broken down into simple parts:

1. The Problem: The Robot is "Blind" to Details

In the world of Artificial Intelligence (AI), there is a task called Aspect-Based Sentiment Analysis (ABSA).

• The Old Way: If a human says, "The pizza was great, but the service was slow," a basic AI might just say, "This review is mixed."
• The New Way (ABSA): A smart AI should say: "The pizza (aspect) was great (opinion) = Positive. The service (aspect) was slow (opinion) = Negative."

The problem? Most of these "smart robots" were trained on English data. When they tried to speak Czech, they were like a tourist who knows a few words but can't understand a complex conversation. Specifically, they were missing the "opinion words" (like delicious, slow, sour) that explain why something was good or bad.

2. The Solution: Building a New "Training Gym"

The authors decided to build a brand new gym (a dataset) specifically for Czech.

• The Dataset: They took 3,000 real Czech restaurant reviews and hired human experts to label them like a game of "Connect the Dots."
• The Connection: They didn't just label "Pizza = Good." They labeled: "Pizza" + "Food Quality" + "Positive" + "Delicious."
• The Result: They created three levels of difficulty for the robots to practice on:
  1. Easy: Find the food and the opinion.
  2. Medium: Add the category (e.g., "Drinks").
  3. Hard: Find everything, even when the food isn't mentioned by name (e.g., "It was tasty" implies the food was tasty, even if the word "food" isn't there).

3. The Experiment: Who Wins the Race?

The authors put the robots through a series of tests to see who could learn Czech best. They tested two types of robots:

• The "Specialist" (Fine-tuned Models): These are smaller, older robots that were trained only on this specific Czech dataset. Think of them as a local chef who has cooked in Prague for 20 years. They know the local ingredients perfectly.
• The "Generalist" (Large Language Models or LLMs): These are massive, super-smart robots (like GPT-4 or LLaMA) that know everything about the world but haven't specifically studied Czech restaurant reviews. Think of them as a world-famous celebrity chef who has never been to Prague but knows how to cook anything.

The Results:

• The Local Chef Wins: The smaller, specialized robots (fine-tuned models) were the most accurate. They understood the nuances of the Czech language best.
• The Celebrity Chef is Close: The massive LLMs did surprisingly well, especially if you gave them a few examples (like showing them a sample menu first). However, they sometimes got confused by subtle Czech idioms or slang.
• The "Translation Trick": The researchers tried a clever hack. They took English reviews, used an AI to translate them into Czech, and then used another AI to fix the labels so they matched the new Czech words. This helped the robots learn faster, acting like a crash course before the real exam.

4. The Hurdles: Where the Robots Stumble

Even the best robots made mistakes. The paper found three main things that trip them up:

1. The "Hidden" Opinion: If someone says, "The waiters were sympathetic," the robot knows "sympathetic" is good. But if they say, "The waiters were sometimes quite sour," the robot gets confused. Is "sour" about the food or the mood?
2. The "Modifier" Trap: In Czech, adding the word "very" (velmi) changes the intensity. "Fast" is good; "Very fast" is great. English datasets often ignore this, but Czech datasets include it. The robots trained on English data often missed this extra layer of meaning.
3. The Idiom Wall: Czech has funny sayings. One example in the paper is "Pivečko jak křen" (literally "Beer like horseradish," meaning the beer is strong and spicy/good). The robots often took this literally and got the sentiment wrong, while the specialized robot got it right because it had seen similar phrases before.

5. The Big Takeaway

This paper is a victory for the Czech language in the AI world.

• For Researchers: They now have a gold-standard dataset to train better models.
• For the Future: They showed that while massive AI models are powerful, specialized training (teaching a robot specifically for a specific job) is still the most reliable way to get perfect results, especially for languages that aren't as common as English.

In a nutshell: The authors built a new, high-quality "textbook" for teaching AI how to understand Czech restaurant reviews. They found that while the "super-smart" AI models are impressive, the "specialized" models that studied the textbook thoroughly are still the best at the job.

Technical Summary

Here is a detailed technical summary of the paper "Extending Czech Aspect-Based Sentiment Analysis with Opinion Terms: Dataset and LLM Benchmarks."

1. Problem Statement

Aspect-Based Sentiment Analysis (ABSA) aims to extract fine-grained sentiment information regarding specific entities and their aspects. While significant progress has been made in English, low-resource languages like Czech face several challenges:

• Lack of Complex Datasets: Existing Czech ABSA datasets (e.g., CsRest-M) support basic triplet extraction (Aspect, Category, Polarity) but lack opinion term annotations. This prevents research on more complex, quadruplet-level tasks such as Aspect Sentiment Triplet Extraction (ASTE), Aspect Sentiment Quad Prediction (ASQP), and Aspect Category Opinion Sentiment (ACOS).
• Cross-Lingual Limitations: Without opinion term annotations, it is impossible to perform robust cross-lingual comparisons or transfer learning with English datasets (like SemEval-2016/ACOS) that include all four sentiment elements.
• Model Performance: The efficacy of Large Language Models (LLMs) versus traditional fine-tuned Transformer models in low-resource, complex ABSA settings remains under-explored, particularly for Czech.

2. Methodology

A. Dataset Construction

The authors extended the existing CsRest-M dataset (3,000 restaurant reviews) to create a new benchmark supporting three tasks:

1. ASTE: Extracting (Aspect, Opinion, Polarity).
2. ASQP: Extracting (Aspect, Category, Opinion, Polarity).
3. ACOS: Extracting (Aspect, Category, Opinion, Polarity), including implicit aspects and opinions.

Annotation Process:

• Opinion Term Addition: Annotators added opinion terms to existing triplets.
• Handling Modifiers: Unlike some English datasets, the Czech dataset explicitly includes modifiers (e.g., "very" in "very fast") within opinion terms to capture sentiment intensity nuances.
• Implicit Terms: Implicit aspects and opinions are marked as NULL.
• Refinement: The annotation guidelines were iteratively refined to resolve ambiguities, achieving a final Inter-Annotator Agreement (IAA) of 85% (strict quadruplet-matching F1).
• Derived Sets: The base ACOS dataset was filtered to create specific ASQP (removing implicit opinions) and ASTE (removing categories and implicit terms) variants.

B. Proposed Cross-Lingual Transfer Method

To address the scarcity of labeled Czech data, the authors proposed a novel Translation and Label Alignment methodology:

1. Translation: English ABSA datasets were translated into Czech using GPT-4o mini.
2. Label Alignment: Instead of simple translation, the LLM was prompted to adjust the labels so that aspect and opinion terms matched the translated Czech text exactly, while keeping categories and polarities unchanged.
3. Validation: Outputs were filtered to ensure the number of tuples and implicit markers matched the source. This approach proved more robust than traditional word-alignment tools (e.g., FastAlign).

C. Experimental Setup

The study evaluated models across three settings: Monolingual, Cross-lingual, and Multilingual.

• Models Tested:
  • Sequence-to-Sequence: Fine-tuned mT5 (multilingual T5).
  • Large Language Models (LLMs): LLaMA 3.1/3.3, Gemma 3, Aya 23, Orca 2, and GPT-4o mini.
• Training Strategies:
  • Zero-shot & Few-shot: Prompt-based inference.
  • Fine-tuning: Full fine-tuning for mT5; QLoRA (4-bit quantization) for LLMs to reduce memory usage.
• Tasks: ASTE, ASQP, and ACOS.
• Metrics: Micro F1-score (strict exact match of all tuple components).

3. Key Contributions

1. New Czech Dataset: The first Czech ABSA dataset with opinion term annotations, supporting quadruplet-level tasks (ACOS/ASQP) and triplet tasks (ASTE). It contains ~7,000 annotated quadruplets across 3,000 sentences.
2. Comprehensive LLM Benchmark: A rigorous evaluation of state-of-the-art LLMs (up to 70B parameters) against fine-tuned baselines in zero-shot, few-shot, and fine-tuned scenarios for Czech ABSA.
3. Cross-Lingual Alignment Framework: A novel pipeline using LLMs to translate and align ABSA labels from English to Czech, enabling effective cross-lingual transfer.
4. Error Analysis: A detailed breakdown of model failures, highlighting specific challenges in Czech linguistics (e.g., implicit opinions, modifier handling).

4. Results

Monolingual Performance

• Fine-tuning Dominance: Fine-tuned models significantly outperformed zero/few-shot LLMs. mT5 achieved the best overall results (Avg F1: 64.30), followed by fine-tuned Aya 23 8B (63.06).
• LLM Gaps: Even the strongest few-shot LLMs (Gemma 3 27B, GPT-4o mini) lagged behind fine-tuned mT5 by 15–25% in F1 score.
• Task Difficulty: Performance decreased as task complexity increased: ASTE (easiest) > ASQP > ACOS (hardest due to implicit terms).

Cross-Lingual Performance

• Translation Impact: Using the proposed translation/alignment method improved cross-lingual performance for several models (e.g., Gemma 3 12B improved by ~4%).
• Limitations: Cross-lingual results remained 15–20% lower than monolingual fine-tuning. A primary cause was the mismatch in handling modifiers (English datasets often omit them, while Czech includes them), causing models trained on English data to miss nuanced Czech sentiment.
• Surpassing Zero-Shot: Cross-lingual fine-tuning significantly outperformed zero-shot and few-shot monolingual baselines (e.g., Orca 2 13B improved by 30% over zero-shot).

Multilingual Performance

• Training on the union of Czech and English data yielded results comparable to monolingual fine-tuning, with mT5 again leading (Avg F1: 64.90).

Error Analysis

• Hardest Element: Opinion terms were the most difficult to predict, followed by aspect terms. Sentiment polarity was the easiest.
• Implicit Terms: ACOS had the highest error rate due to the difficulty of detecting implicit opinions.
• Model Behavior: Fine-tuned mT5 produced fewer errors and adhered better to dataset conventions. LLMs (especially zero-shot) struggled with:
  • Idioms (e.g., misclassifying a positive Czech idiom about beer as negative).
  • Formatting and generation consistency.
  • Over-generalizing aspect terms (e.g., grouping specific foods into a generic "food" category).

5. Significance and Conclusion

This paper establishes a new standard for Czech ABSA research by providing a high-quality, opinion-rich dataset. It demonstrates that while fine-tuned models (specifically mT5) remain the most reliable and efficient solution for low-resource ABSA tasks, LLMs offer viable flexible alternatives, especially when combined with few-shot prompting or cross-lingual transfer.

The proposed LLM-based translation and label alignment method offers a scalable solution for adapting ABSA resources to other low-resource languages, reducing the need for expensive manual annotation. However, the study highlights that linguistic nuances (such as modifiers and idioms) and dataset inconsistencies remain significant barriers to perfect cross-lingual transfer, pointing to future research directions in robust multilingual modeling.