GitBugs: Bug Reports for Duplicate Detection, Retrieval Augmented Generation, Triage, and More

1. Problem Statement

Despite the critical role of bug reports in software maintenance and quality assurance, existing datasets for software engineering research suffer from significant limitations:

• Scope and Breadth: Many datasets are restricted to single projects or specific ecosystems (e.g., only Eclipse or only Java).
• Data Quality: Existing resources often lack standardized metadata, are outdated, or contain unstructured data unsuitable for modern machine learning (ML) and Large Language Model (LLM) applications.
• Reproducibility: There is a lack of consistent train/test splits and standardized benchmarks for tasks like duplicate detection and triaging across diverse platforms.
• Metadata Gaps: Critical fields required for supervised learning (e.g., precise resolution times, duplicate mappings, priority labels) are often missing or inconsistent.

2. Methodology

The authors developed GitBugs, a large-scale, curated dataset designed to overcome these limitations through a rigorous data collection and standardization process.

• Data Sources: The dataset aggregates bug reports from nine major, actively maintained open-source projects spanning diverse domains (distributed systems, browsers, IDEs, cloud infrastructure):
  • Projects: Cassandra, Firefox, Hadoop, HBase, Mozilla Core, VS Code, SeaMonkey, Spark, and Thunderbird.
  • Platforms: Data was harvested from GitHub, Jira, and Bugzilla issue trackers.
• Collection Methodology:
  • Acquisition: Used a hybrid approach involving RESTful API calls (for Jira-based systems like Apache projects) and HTML scraping/CSV archive parsing (for Bugzilla-based systems like Firefox and Thunderbird).
  • Filtering: Reports were filtered to exclude non-bug entries (e.g., feature requests, tasks) based on metadata fields like "issue type."
  • Standardization: Key categorical fields (Summary, Description, Status, Priority, Resolution, Timestamps) were normalized across different tracker schemas to create a unified schema.
• Dataset Composition:
  • Volume: Over 150,000 bug reports.
  • Annotations: Includes explicit mappings for duplicate bug reports, enabling supervised learning for duplicate detection.
  • Splits: Predefined train/test splits are provided to ensure reproducible benchmarking.
  • Artifacts: The release includes exploratory data analysis (EDA) notebooks, model training scripts, and validation tools.

3. Key Contributions

1. Large-Scale, Multi-Source Dataset: A corpus of 150k+ reports from 9 projects across GitHub, Jira, and Bugzilla, offering a level of diversity previously unavailable in a single resource.
2. Rich Metadata & Standardization: Comprehensive fields including timestamps, priority, resolution status, and duplicate links, standardized for immediate use in ML pipelines.
3. Project-Level Analytics: Built-in statistics on duplicate rates (ranging from 2.0% in HBase to 28.2% in VS Code) and resolution time distributions.
4. Reproducible Research Artifacts: The authors provide the code, data mappings, and EDA notebooks to facilitate immediate experimentation.
5. Benchmarking for Emerging Tasks: The dataset is specifically structured to support not only traditional tasks (triage, severity prediction) but also modern LLM-based tasks like Retrieval-Augmented Generation (RAG) and automated bug explanation.

4. Results and Case Studies

The authors conducted a comprehensive case study using the Apache Cassandra subset to evaluate the dataset's utility across various tasks:

• Bug Volume Forecasting:
  • Models: ARIMA vs. Prophet.
  • Result: ARIMA outperformed Prophet (MAE: 10.92 vs. 20.10), demonstrating better stability in forecasting bug submission volumes.
• Severity/Priority Classification:
  • Result: The model achieved high accuracy (82%) on the dominant "Normal" class but struggled with minority classes (Macro F1: 0.35), highlighting significant class imbalance challenges inherent in bug data.
• Time-to-Fix Prediction:
  • Result: Regression models performed poorly (R² = -0.09), often underestimating long resolution times. This indicates that textual signals alone are insufficient for predicting fix duration without more complex features.
• Duplicate Detection:
  • Method: Cosine similarity using Sentence-BERT embeddings.
  • Result: Achieved a Recall@10 of 0.61. However, similarity scores were generally low (<0.5), indicating that duplicate bugs are often linguistically subtle and require semantic understanding rather than simple keyword matching.
• Retrieval-Augmented Generation (RAG):
  • Demonstration: A pipeline was built to retrieve semantically similar historical reports to generate enriched bug explanations.
  • Outcome: The system successfully used historical context to hypothesize root causes (e.g., missing event listeners), proving the dataset's viability for training LLMs on software maintenance tasks.
• Temporal Analysis:
  • Topic modeling (LDA) and STL decomposition revealed seasonal trends aligned with release cycles and a long-term decline in bug reports, suggesting increasing system maturity.

5. Significance and Impact

• Reproducibility: GitBugs addresses the "reproducibility crisis" in empirical software engineering by providing standardized, open-access data with clear train/test splits.
• Bridge to Modern AI: By structuring data for both traditional ML and LLM applications, it facilitates research into Retrieval-Augmented Generation (RAG) for debugging, automated triaging, and bug report summarization.
• Industry Applicability: The dataset serves as a benchmark for internal tools (deduplication engines, triage recommenders) and a resource for fine-tuning LLMs on realistic software engineering data.
• Educational Value: It provides a hands-on resource for teaching software analytics, allowing students to work with real-world, messy, yet structured bug data.

Availability: The dataset and associated code are publicly available at https://github.com/av9ash/gitbugs/.