GitReq: A Gold Standard Dataset for Software Quality Requirements
This paper introduces GitReq, a publicly available dataset of 6,302 expert-validated GitHub issues categorized into eight ISO/IEC 25010:2011-aligned software quality requirements, which serves as a gold standard for advancing automated requirement classification and software quality analysis.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are walking into a massive, chaotic library where millions of people have left sticky notes on the shelves. These notes aren't just random complaints; they are the developers of the world's software whispering (or shouting) about how their creations should work better. Some say, "This app is too slow!" (Performance). Others say, "We need a better lock on the door!" (Security). Some say, "It should work on my old phone, too!" (Portability).
The problem is, these notes are a mess. They are mixed together, written in slang, and buried under millions of other notes about bugs, questions, and feature requests. Until now, no one had organized these specific "quality" notes into a neat, labeled collection that researchers could study.
Enter GitReq: The Great Librarian.
The authors of this paper built GitReq, a "Gold Standard" dataset. Think of it as a meticulously organized archive of 6,302 of these developer notes, pulled from over 4,000 different software projects on GitHub.
Here is how they did it, broken down into simple steps:
1. The Treasure Hunt (Mining)
The team didn't just grab random notes. They used a "three-signal" strategy to find the right ones. Imagine looking for a specific type of fish in a pond:
- Signal 1: They looked for the right "flag" (a label the developer already put on the note, like "Security").
- Signal 2: They checked if the note was actually a request for something new (labels like "Feature Request" or "Enhancement"), ignoring notes that were just bug reports or questions.
- Signal 3: They scanned the text for specific keywords (like "slow," "hack," or "crash").
They started with 55,588 potential notes. That's a huge pile of paper!
2. The Sorting Machine (Preprocessing)
Before the humans could read them, the team built two different "sorting machines" because the notes came in two very different flavors:
- The "NFR" Machine (Non-Functional Requirements): These are notes about how the system behaves (speed, safety, reliability). These notes are often short, messy, and slang-heavy (e.g., "Server crashes when 100 people log in"). The machine cleaned up the noise but kept the messy, real-world language.
- The "FR" Machine (Functional Requirements): These are notes about what the system should do (e.g., "The system must allow users to save files"). These need to be very specific. The machine was strict: if the note didn't sound like a formal rule (using words like "must," "shall," or "user story"), it was thrown out. This ensured they didn't accidentally include vague feature ideas.
3. The Expert Panel (Human Annotation)
After the machines did their work, they still had about 8,500 notes. This is where the human magic happened.
- The Judges: Seven experts in software engineering sat down to read these notes.
- The Training: They spent hours learning the rules, using a standard guide called ISO/IEC 25010. Think of this as a rulebook that defines exactly what "Security" means versus "Scalability."
- The Verdict: They labeled each note into one of eight categories: Performance, Security, Portability, Availability, Fault-tolerance, Scalability, Maintainability, and Functional.
- The Agreement: They didn't just guess. They checked their work against each other. When they disagreed, they talked it out until they agreed. The result was a very high level of agreement (a score of 0.72), meaning the labels are trustworthy.
4. The Final Collection
From the original 55,000+ candidates, they ended up with 6,302 high-quality, expert-verified notes.
- The Mix: About half are about Security and Performance (the most common worries).
- The Variety: They cover everything from web frameworks to mobile apps and cloud systems.
- The Proof: They even tested this new dataset against four powerful AI models (like GPT-5.2). The AI models struggled a bit, especially with tricky categories like "Maintainability," proving that this dataset is a tough, realistic test for future AI tools.
Why Does This Matter?
Before this, researchers trying to teach computers to understand software quality had to use tiny, old datasets or formal documents that didn't look like real life. It was like trying to learn how to drive a car by only reading a textbook written in 1990.
GitReq is like giving researchers a brand-new, real-world driving simulator. It allows them to build better AI tools that can actually understand the messy, real conversations developers have every day, helping to spot quality issues faster and more accurately.
In short: The paper didn't just find a needle in a haystack; they built a whole new library of needles, sorted them by type, and gave the world a map to find them.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.