Evaluating Language Model Applications for Identifying Solution-Related Content in Issue Report Discussions

1. Problem Statement

Software developers rely on issue reports (e.g., in Bugzilla, Jira, GitHub) to discuss, design, and implement solutions for defects, features, and enhancements. These discussions are often lengthy, containing hundreds of comments with mixed content (problem descriptions, code reviews, questions, and solution proposals).

• The Challenge: Manually locating "solution-related content" (proposed solutions, solution assessments, and implementation details) within these discussions is time-consuming and cognitively demanding.
• The Gap: While text classification has been used to identify various software artifacts, it remains unclear how different classes of models—Traditional Machine Learning Models (MLMs), Pre-trained Language Models (PLMs), and Large Language Models (LLMs)—perform on this specific task. Furthermore, the efficacy of combining these models and their ability to generalize across different software projects is under-explored.

2. Methodology

The authors conducted an empirical study using 356 Mozilla Firefox issue reports, comprising 4,917 manually annotated comments (808 solution-related, 4,109 non-solution-related).

2.1 Experimental Design

The study evaluated 12 distinct classifiers across 68 configurations, categorized by three model families and three application strategies:

1. Machine Learning Models (MLMs): Six traditional classifiers (Logistic Regression, KNN, Naïve Bayes, SVM, Decision Trees, Random Forest).
   • Input: Tested with Tf-Idf (baseline) and embeddings from BERT, Llama-3, and GPT-4.
2. Pre-trained Language Models (PLMs): Four models (BERT, DistilBERT, RoBERTa, XLNet).
   • Strategy: Fine-tuning for binary classification.
3. Large Language Models (LLMs): Two models (Llama-3-8B, Llama-3-70B).
   • Strategy A (Prompting): Llama-3-70B used for Zero-shot, Few-shot, and Chain-of-Thought (CoT) prompting.
   • Strategy B (Fine-tuning): Llama-3-8B fine-tuned using LoRA (Low-Rank Adaptation) and 4-bit quantization.

2.2 Evaluation Metrics

• Primary Metric: F1 Score (harmonic mean of Precision and Recall).
• Secondary Metrics: Precision, Recall, and Relative Improvement (RI) over baselines.
• Ensemble Strategy: Majority voting (conservative and non-conservative) combining top performers from each model family.
• Generalizability: Models trained on Mozilla were tested on Chromium and GnuCash datasets under three settings: Project-Specific, Cross-Project, and Hybrid (Mozilla + Target Project data).

2.3 Qualitative Analysis

The authors performed a manual analysis of misclassifications (False Positives and False Negatives) by the top three models (SVM, RoBERTa, Llama-3ft) to identify failure patterns.

3. Key Contributions

1. Comprehensive Benchmarking: The first study to systematically compare MLMs, PLMs, and LLMs (via embeddings, prompting, and fine-tuning) for identifying solution content in issue discussions.
2. Ensemble Analysis: Evaluation of combining diverse model families to enhance prediction reliability.
3. Qualitative Insights: Detailed analysis of why models fail, focusing on misleading clues (keywords, code snippets) and lack of context.
4. Cross-Project Generalizability: A study demonstrating how knowledge transfers from Mozilla to other projects (Chromium, GnuCash) and the impact of hybrid training.
5. Replication Package: Full release of source code, data, and documentation.

4. Key Results

4.1 Model Performance (Mozilla Dataset)

• Fine-Tuning is Superior: Fine-tuned models significantly outperformed prompting and traditional MLMs with Tf-Idf.
  • Best Single Model: Fine-tuned Llama-3 (Llama-3ft) achieved the highest F1 score of 0.716 (Precision: 0.626, Recall: 0.838).
  • Runner-up: Fine-tuned RoBERTa achieved an F1 of 0.692.
• MLMs with LM Embeddings: Traditional models using LLM embeddings (specifically SVM with GPT-4 embeddings) performed well (F1: 0.663), outperforming Tf-Idf baselines. This suggests lightweight classifiers can leverage rich semantic representations without full fine-tuning.
• Prompting Underperformed: Llama-3 via prompting (even with Chain-of-Thought) achieved a maximum F1 of 0.517. Including issue descriptions in prompts actually degraded performance by overloading the model with context.
• Ensemble Success: An ensemble of the best models from each category (SVM + RoBERTa + Llama-3ft) achieved the highest overall F1 of 0.737, a 2.9% improvement over the best single model.

4.2 Generalizability (Cross-Project)

• Transferability: Models fine-tuned on Mozilla data generalized well to Chromium and GnuCash, outperforming models trained only on small project-specific datasets.
• Hybrid Training: The best results were achieved by Hybrid Training (Mozilla data + small amount of target project data).
  • RoBERTa showed the strongest adaptability in the generalization study (F1: 0.961 on GnuCash), outperforming Llama-3ft on smaller datasets.
  • Llama-3ft required more project-specific data to adapt effectively compared to smaller PLMs.

4.3 Qualitative Findings

• False Positives: Often caused by comments containing technical jargon, code snippets, or keywords like "fix" or "patch" that referred to reviews or debugging rather than proposing a solution.
• False Negatives: Often caused by short, abstract, or implicit solution descriptions (e.g., "This fixes the issue") that lacked sufficient context or explicit phrasing.
• Context Dependency: Models struggled when comments were isolated from the issue description or surrounding discussion threads.

5. Significance and Implications

• Practical Tooling: The study validates that automated tools can effectively highlight solution-related comments, aiding developers in debugging regressions, reusing solutions, and understanding code rationale.
• Resource Efficiency: Teams with limited computational resources can achieve strong results using SVM with GPT-4 embeddings (F1 0.663) without the cost of fine-tuning massive LLMs.
• Deployment Strategy: For new projects, a Hybrid Training approach is recommended: start with a model pre-trained on a large dataset (like Mozilla) and fine-tune it with a small, curated set of project-specific issues to maximize performance.
• Future Directions: The study highlights the need for context-aware classifiers that analyze entire discussion threads rather than isolated comments to resolve ambiguity and improve accuracy.

In conclusion, this paper establishes that while fine-tuned LLMs offer the highest accuracy for solution identification, a combination of diverse model architectures and hybrid training strategies provides the most robust and generalizable solution for software maintenance tasks.