Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?

1. Problem Statement

Sentiment analysis in Software Engineering (SE) is crucial for understanding developer emotions, user feedback, and community dynamics. However, existing approaches face significant hurdles:

• Domain Specificity: General-purpose sentiment tools often fail in SE contexts (e.g., code reviews, API documentation) due to unique jargon and context.
• Data Scarcity: Developing specialized tools requires supervised machine learning, which relies on large, annotated datasets. Creating these datasets is costly, time-consuming, and requires domain expertise.
• Context Boundaries: Models trained on one SE context (e.g., Stack Overflow) often fail to generalize to others (e.g., GitHub pull requests), necessitating separate models for each context.

The authors investigate whether Zero-Shot Learning (ZSL) can overcome the dependency on annotated datasets by classifying sentiment in unseen SE contexts without prior task-specific training.

2. Methodology

The study employs an empirical experimental design to evaluate ZSL techniques across diverse SE datasets.

Datasets

Seven publicly available datasets were utilized, covering various SE artifacts:

• API Reviews: Stack Overflow (SO) posts.
• Code Reviews: Gerrit comments.
• Collaboration: GitHub pull requests/commits, Gitter messages.
• User Feedback: Google Play app reviews.
• Issue Tracking: Jira issue comments.
• Technical Q&A: Stack Overflow posts.

ZSL Techniques Evaluated

The study compared four distinct ZSL paradigms:

1. Embedding-based: Measures semantic similarity between input text and label embeddings using Cosine similarity (e.g., BERT, RoBERTa, XLNet).
2. Natural Language Inference (NLI)-based: Frames classification as an entailment problem (Premise: Text vs. Hypothesis: Label) to determine logical consistency.
3. Task-Aware Representation of Sentences (TARS): Formulates the task as a binary classification problem for each label, adapting representations via label-conditioned embeddings.
4. Generative-based: Uses Large Language Models (LLMs) like GPT-3.5 Turbo to generate the sentiment label directly from a natural language prompt.

Label Configurations

To test the impact of prompt engineering, three label types were curated:

• Original (L1): Standard dataset labels (e.g., "Positive").
• Expert-Curated (L2–L5): Human-designed labels incorporating context (e.g., "An app review with positive sentiment") and mapped emotions.
• LLM-Generated (L6–L7): Labels generated by ChatGPT, including lists of descriptive words (e.g., "cheerfulness, happiness...").

Baseline Comparison

The best-performing ZSL configurations were compared against State-of-the-Art (SOTA) fine-tuned transformer models (BERT, RoBERTa, etc.) trained on 80% of the data, with evaluation on the remaining 20%.

Metrics & Analysis

• Performance: Macro-F1 and Micro-F1 scores.
• Statistical Significance: Scott-Knott Effect Size Difference (ESD) test to group models into statistically distinct performance tiers.
• Error Analysis: Quantitative and qualitative analysis of misclassifications using a framework adapted from Novielli et al.

3. Key Contributions

1. Comprehensive ZSL Evaluation: The first extensive empirical study comparing embedding, NLI, TARS, and generative ZSL techniques specifically for SE sentiment analysis.
2. Label Configuration Impact: Demonstrates that expert-curated labels (specifically those adding context like "An app review with...") significantly boost ZSL performance, often outperforming original dataset labels.
3. Benchmarking against Fine-Tuning: Provides evidence that specific ZSL models can match or exceed the performance of models requiring full fine-tuning, challenging the assumption that supervised learning is always superior in data-scarce domains.
4. Error Taxonomy: Identifies specific failure modes in SE sentiment analysis, such as "polar facts" (neutral statements about desirable/undesirable outcomes) and annotation subjectivity.

4. Key Results

Performance Rankings (RQ1 & RQ2)

• Best Technique: Generative-based ZSL (GPT-3.5) and NLI-based ZSL generally outperformed others.
• Best Model: E M9 (Twitter-RoBERTa-base-sentiment), an embedding-based model pre-trained on social media, achieved the highest Macro-F1 scores across most datasets when paired with expert-curated labels.
• Best Label Configuration: L3 (Expert-curated: "An [instance type] with [sentiment] sentiment") yielded the best results for embedding and NLI models.
• Top Combination: E M9 + L3 achieved the highest overall performance, statistically ranking as the top performer.

Comparison with Fine-Tuned Models (RQ3)

• Competitive Parity: The best ZSL configuration (E M9 + L3) achieved performance comparable to SOTA fine-tuned models on the test sets.
• Contextual Nuance:
  • Fine-tuned models excelled on technical, structured data (API reviews, code reviews, GitHub comments, SO posts).
  • ZSL models performed surprisingly well, and sometimes better, on conversational and user-generated data (Gitter, App reviews, Jira).
• Conclusion: ZSL eliminates the need for expensive data annotation while maintaining high accuracy, making it a viable alternative for new SE contexts.

Error Analysis (RQ4)

• Primary Failure Mode: The Neutral class was the most frequently misclassified (64.71% of ZSL errors).
• Root Causes:
  1. Subjectivity in Annotation (60.29%): Different human annotators perceive emotions differently.
  2. Polar Facts (22.06%): Statements describing inherently good/bad situations in a neutral tone (e.g., "This doesn't work") confuse models.
  3. Politeness & Figurative Language: Polite phrases ("Thanks!") and sarcasm/irony caused misclassifications.
• Shared Errors: Both ZSL and fine-tuned models struggled with the same 9 API reviews, 5 GitHub comments, and 3 app reviews, suggesting these are inherently ambiguous cases.

5. Significance and Implications

• Solving Data Scarcity: This study proves that SE researchers and practitioners do not always need to invest in creating large, annotated datasets. ZSL offers a "plug-and-play" solution for sentiment analysis in new or niche SE contexts.
• Cost Efficiency: By utilizing pre-trained models (especially free, open-source ones like Twitter-RoBERTa) and expert-curated prompts, organizations can avoid the high costs of data labeling and model fine-tuning.
• Practical Guidelines: The paper provides actionable advice:
  • Use expert-curated labels that explicitly describe the context and sentiment.
  • Prioritize embedding-based models pre-trained on similar domains (e.g., social media) over generic models.
  • Be aware that neutral sentiment and polar facts remain the hardest challenges for current AI models.

In summary, the paper establishes that Zero-Shot Learning is a robust, effective, and cost-efficient approach for sentiment analysis in software engineering, capable of rivaling traditional supervised methods without the burden of data annotation.