Entity and non-functional-requirement extraction from user stories: evidence from supervised sequence models and reproducible LLM benchmarking
This paper evaluates supervised sequence models and open-weight LLMs for extracting entities and non-functional requirements from user stories, demonstrating that a BiLSTM-CRF model significantly outperforms zero-shot and one-shot LLMs on a newly released, reproducible benchmark.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Software teams often write their plans as short, natural sentences called user stories. These notes follow a simple pattern: "As a [person], I want [action], so that [reason]." They are meant to capture what a piece of software should do and why it matters. While humans can read these notes easily, computers struggle to understand them. The sentences are often messy, missing parts, or written in a way that doesn't fit a strict template. To build better tools for managing software, researchers need a way to teach computers to read these stories and pull out the specific pieces of information: who is involved, what they want to do, and why they want it. They also need to spot hidden requirements, such as rules about speed, privacy, or how easy the system is to use, which are often buried in the text or left unsaid.
A team of researchers set out to solve this problem by testing two different approaches to teaching computers how to extract these details. The first approach uses a specialized type of computer program trained on thousands of examples to act like a careful editor, highlighting the exact words that fit each category. The second approach uses large language models, which are powerful artificial intelligence systems capable of understanding and generating human language without needing specific training for the task. The researchers wanted to see which method could more accurately pull the correct information from a large collection of software stories, and whether the newer, more flexible AI tools could outperform the older, more rigid ones.
The team began by gathering a massive collection of 2,234 English user stories from various public sources, including software development communities and requirement databases. They labeled these stories by hand, marking exactly which words represented the person, the action, and the goal. They then tested a series of computer models on this data. The most successful model was a system that reads text in both directions at once, allowing it to understand the context of a word by looking at the words before and after it. This model, when trained on the original collection, correctly identified the key parts of the stories 96.2% of the time. When the researchers added more examples to the training data by creating slight variations of the existing stories, the model's accuracy improved to 98.6%. This system also proved very good at spotting the five specific quality aspects the team was looking for: privacy, performance, usability, scalability, and maintainability.
To ensure their results were fair and could be checked by others, the researchers created a new, shared test set that was kept separate from the training data. They invited seven different large language models to try the same task. These models were given the stories and asked to return the answers in a strict format, without any extra explanation. The researchers measured success by checking if the computer's answer matched the human-labeled answer word-for-word and in the exact same order. The results showed a clear gap between the two types of technology. The specialized computer program trained on the specific data achieved an accuracy score of 0.953 on the new test set. Even when the researchers added a small number of safe, modified examples to its training, the score rose to 0.961.
In contrast, the large language models, even the most powerful ones available, struggled significantly when asked to match the exact boundaries of the words. The best-performing large language model without any specific examples to learn from got only 0.414. When the researchers gave that same model one example to show it how to do the task, its score improved slightly to 0.434. The main issue was that while these models often understood the general meaning of the story, they frequently included too many or too few words in their answers, or they picked the wrong starting and ending points for the phrases. Because the test required an exact match, these small errors counted as failures. The specialized program, however, was designed specifically to find those exact boundaries, making it far more reliable for this specific job.
The study also looked at how fast these systems worked. The specialized program could process a single story in less than half a millisecond on a standard computer, making it incredibly fast for large collections of data. The large language models took much longer, averaging about 0.7 seconds per story, which is a significant difference when dealing with thousands of documents. The researchers noted that while the large language models are excellent at understanding the general idea and can be helpful for a human to review quickly, they are not yet ready to replace the specialized programs for tasks that require precise, automated extraction of specific details.
The researchers were careful to point out the limits of their work. They could not fully reproduce the results of an earlier, larger experiment because some of the original data and logs were missing. They also noted that the shared benchmark they created intentionally left the quality aspect (NFR) labels empty because the available source data was not validated for story-level categories. Consequently, this new shared test set does not allow for a fair comparison of how well the large language models handled those specific non-functional requirements. The study concludes that for teams that need to automatically sort through thousands of software stories to find specific actors, actions, and goals, the specialized computer program remains the most accurate and efficient tool. The large language models can serve as a helpful assistant to a human analyst, but they cannot yet be trusted to do the work alone without careful checking. The researchers have released their data and tools so that other scientists can verify these findings and continue to improve the technology.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.