User eXperience Perception Insights Dataset (UXPID): Synthetic User Feedback from Public Industrial Forums

Technical Summary: The User eXperience Perception Insights Dataset (UXPID)

Problem Statement
Customer feedback in industrial technical forums offers critical insights into real-world product experiences, usability issues, and feature requests. However, systematic analysis of this feedback is hindered by the unstructured nature of the content, the scarcity of high-quality labeled datasets, and significant privacy/licensing restrictions that prevent the use of real-world data for training AI models. Existing open-access datasets for requirements engineering and user feedback analysis (e.g., TFThs, TechQA, PURE) often lack contextual discussion structures, severity annotations, or semantic insights derived from conversational threads. Furthermore, many are too small for robust model training or fail to capture the specific nuances of industrial automation contexts, limiting the development of models capable of extracting actionable requirements and UX insights at scale.

Methodology
The authors present the User eXperience Perception Insights Dataset (UXPID), a synthetic dataset constructed from 7,130 discussion branches extracted from a public industrial automation forum. The dataset creation pipeline involves four key stages:

1. Data Collection and Extraction: User comments and metadata were collected from a company-operated technical forum. The data was stored in JSON format, preserving the structure of multi-post comment threads.
2. LLM-Driven Annotation and Synthesis: The OpenAI GPT-4.1 model was utilized to process the data. Crucially, the process was sequential to avoid circularity:
   • Step 1 (Annotation): The LLM analyzed the original, unmodified forum text to extract semantic insights, including UX summaries, user expectations, severity ratings, sentiment, and keyword extraction (pain, gain, feature). These annotations serve as the ground truth.
   • Step 2 (Synthesis/Anonymization): The LLM was then instructed to lightly rephrase the original text and replace all identifying entities (company names, product names, user names, URLs, etc.) with typed placeholders (e.g., [product_name], [user_name]). This ensures privacy while preserving the semantic content required for the annotations.
3. Topic Classification: An iterative process was applied where the LLM independently extracted topics for each branch. New topics were added to a storage pool if they did not match existing ones, allowing for dynamic topic clustering.
4. Human Validation: To ensure annotation accuracy, 32 purposively selected branches were manually validated by three domain experts against 20 annotation fields. The validation covered use cases, UX metrics, severity, and sentiment. The process achieved an 87% "adequate-agreement" rate (combining exact and minor disagreements), with prompt refinements reducing systematic discrepancies between validation rounds.
5. Fidelity Analysis: A statistical comparison (Wilcoxon signed-rank test) between original and synthesized text confirmed that conversational structure (comment counts, reply ratios) was preserved. While the synthesis process increased surface formality (reducing informal markers like exclamation points and ALL-CAPS) and slightly altered word counts, the core linguistic diversity and interrogative structure remained largely intact.

Key Contributions
The paper identifies three primary contributions:

1. A Novel Dataset (UXPID): A collection of 7,130 anonymized, synthesized forum branches containing over 36,000 comments, specifically tailored to industrial automation products. It extends the scope of existing requirements engineering datasets by capturing full conversational contexts.
2. Rich Semantic Structure: Unlike prior resources, UXPID includes automated insights, topic clustering, severity annotations, and sentiment analysis derived from user conversations, providing contextual depth for training NLP models.
3. Benchmark Baselines: The authors provide reference performance metrics for classification tasks using both classical (TF-IDF + Logistic Regression) and transformer-based (DistilBERT) models, establishing a baseline for future research in feedback classification and requirement extraction.

Results
Technical validation was performed using classification experiments on a stratified 80/20 train/test split.

• Topic Classification: DistilBERT consistently outperformed the TF-IDF + Logistic Regression baseline. For "Summary → Topic" classification across all 32 classes, DistilBERT achieved an F1 score of 0.6923 compared to 0.4799 for the baseline. When restricted to classes with >50 instances, the gap narrowed (0.7174 vs. 0.6060), suggesting the baseline struggles primarily with rare classes.
• Sentiment Analysis: DistilBERT achieved F1 scores of 0.7825 (summary input) and 0.7923 (raw comment input) for overall sentiment classification, significantly outperforming the baseline (0.6568 and 0.6340, respectively).
• Model Behavior: The transformer model predicted label counts (1.53–1.62 per sample) that closely matched the ground truth (1.69), whereas the classical baseline tended to over-predict labels.

Significance and Claims
The paper claims that UXPID addresses the critical gap in available resources for training AI models on industrial user feedback. By providing a privacy-preserving, well-annotated dataset, it enables the training and evaluation of transformer-based models for tasks such as issue detection, sentiment analysis, and requirements extraction in technical forums. The authors position the dataset as a valuable resource for advancing NLP methods within industrial product support and software engineering, particularly where access to real-world data is restricted. The dataset is designed to support diverse applications in requirements engineering and natural language processing, offering a foundation for developing more efficient strategies to analyze large volumes of unstructured industrial feedback.