User Reviews as a Source for Usability Requirements: A Precursor Study on Using Large Language Models

Technical Summary: User Reviews as a Source for Usability Requirements Using Large Language Models

Problem Statement
Modern software products face increasing functional complexity, creating significant usability challenges. While user-centered requirements engineering (RE) is known to improve product suitability, directly involving users is resource-intensive and often unfeasible for development teams. Crowd-based Requirements Engineering (CrowdRE) attempts to bridge this gap by leveraging public user feedback (e.g., app store reviews). However, existing approaches for classifying usability aspects from this feedback rely heavily on Machine Learning (ML) or Deep Learning (DL). These traditional methods require extensive, manually labeled training datasets, creating a bottleneck for rapid development cycles. Furthermore, standard ML models often struggle with contextual nuances and informal language (e.g., emojis, idioms) common in user reviews, typically relying on simple keyword matching. There is a critical research gap in combining the advanced natural language understanding of Large Language Models (LLMs) with the user-centric methods of CrowdRE to extract usability requirements without the need for intensive task-specific training.

Methodology
The study investigates whether a specifically prompted LLM (OpenAI's gpt-4.1) can effectively identify usability-relevant user reviews comparable to human raters. The research is structured around two Research Questions (RQs):

1. Is the performance of a tailored LLM comparable to human coders in identifying usability-related reviews?
2. What is the reliability of the LLM in identifying these reviews within a random dataset?

Data Collection and Preparation:

• Dataset: A dataset of 300 user reviews was collected from the Google Play Store, evenly distributed across three distinct application types to ensure diversity: BlitzerDE (maps/navigation), Lidl Plus (e-commerce), and FL Studio (music composition).
• Human Labeling (Ground Truth): Two experienced raters labeled the dataset based on the 10 Nielsen Usability Heuristics. Reviews were coded as "true" (usability-relevant) if they contained at least one statement related to a heuristic, and "false" otherwise.
  • Exclusions: Feature requests, overlapping non-functional requirements (e.g., accessibility, compatibility), vague statements, bug reports, and issues stemming from external contexts (e.g., OS or hardware) were excluded.
  • Process: An iterative coding process was employed. A pre-coding round on 20 reviews identified guideline ambiguities, leading to an enhanced coding guideline. A second round covered the full 300 reviews. Inter-rater reliability was calculated using Cohen's Kappa ($\kappa$), showing substantial agreement across all three apps ($\kappa$ ranging from 0.61 to 0.76).
• LLM Prompt Engineering: An initial prompt was developed based on the enhanced coding guidelines using Chain-of-Thought prompting to instruct the LLM on intermediate reasoning steps. The prompt was refined through three iterations using a test sample of 30 reviews (10 per app). The authors explicitly avoided few-shot prompting to prevent overfitting, instead refining the instructions and goals within the prompt.

Key Contributions
The paper provides three primary contributions:

1. A Coded Dataset: A fully labeled dataset of 300 user reviews from three diverse app categories, labeled by two human raters and an LLM, available in supplementary material.
2. Prompt Engineering Framework: An initial prompt and coding guidelines specifically derived from the 10 Nielsen Usability Heuristics to filter usability-relevant reviews, developed through iterative refinement.
3. Performance Analysis: An empirical determination of the LLM's capability to recognize usability as a non-functional requirement compared to human performance.

Results

• Human Baseline: The two human raters identified 148 (49.3%) of the 300 reviews as usability-relevant. Inter-rater reliability was substantial ($\kappa$ 0.61–0.76).
• LLM Performance (Metrics): Using the final optimized prompt, the LLM identified 173 (58%) reviews as usability-relevant.
  • Precision/Recall: The LLM demonstrated high Recall (0.84–0.93 across apps) but slightly lower Precision (0.73–0.79), indicating it is more permissive than human raters (identifying more potential usability issues, some of which humans rejected).
  • F-Score: The LLM achieved F1-scores between 0.78 and 0.82 across the different apps.
• Reliability (Inter-rater Agreement): When comparing the LLM's labels against the final human consensus:
  • BlitzerDE: Fair agreement ($\kappa = 0.34$).
  • Lidl Plus: Moderate agreement ($\kappa = 0.55$).
  • FL Studio: Substantial agreement ($\kappa = 0.63$).
• Statistical Significance: A Chi-square test on the correlation between rater certainty and LLM correctness yielded a p-value of 0.35, indicating no statistically significant correlation. The LLM does not reliably reproduce the human mental model and makes errors in different, unexpected places.

Significance and Claims
The paper concludes that while LLMs are not yet perfect replacements for human raters due to lower reliability and agreement levels, they represent a viable and resource-efficient option for requirements engineering.

• Trade-off: The authors argue that the resource overhead of training traditional ML/DL models is impractical for most teams. In contrast, LLMs achieve acceptable results with significantly less overhead.
• Permissiveness: The LLM's tendency to be slightly more permissive (higher recall) is viewed as a practical benefit; in a real-world scenario, gathering a limited amount of irrelevant reviews is preferable to missing relevant usability requirements.
• Future Direction: The study serves as a precursor, suggesting that further research should focus on sophisticated prompt engineering (e.g., few-shot prompting) and comparing optimized LLMs against traditional ML/DL approaches to determine the optimal technology for specific CrowdRE tasks. The authors emphasize that the success of the approach is heavily dependent on the quality of the tailored prompt.