🤖 AI

Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback

This paper presents techniques for efficiently analyzing large volumes of user feedback by combining supervised multi-label classification for topic categorization with generative AI for summarization, while demonstrating that sentiment analysis is an unreliable proxy for overall product satisfaction, which must be measured explicitly.

Original authors: Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp

Published 2026-06-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sandra Loop, Erik Bertram, Sebastian Juhl, Martin Schrepp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, bustling restaurant. Every night, hundreds of customers leave feedback cards in a suggestion box. Some cards say, “The soup was delicious!” while others write three paragraphs about how the table wobbled and the music was too loud.

If you try to read every single card by hand, you’ll burn out before lunch. If you just look at the star ratings (1 to 5 stars), you might see a 4.5 average and think everything is perfect, missing the fact that 20 people complained about broken chairs.

This paper describes how a large software company (SAP) solved this exact problem for their digital products. They built a system to automatically sort, summarize, and interpret thousands of user comments so managers can actually understand what’s going on without reading every word.

Here is how their system works, broken down into three main parts:

1. The Smart Sorting Hat (Topic Classification)

First, they needed to organize the chaos. Instead of reading comments, they taught a computer to act like a librarian. They created a list of specific "topics" or labels, such as Usability (is it easy to use?), Performance (is it slow?), Errors (did it crash?), and Visual Design (does it look good?).

How it works: The computer reads each comment and tags it with one or more of these labels. For example, if a user writes, “The app is beautiful but crashes when I click save,” the computer tags it as both Visual Design (positive) and Error (negative).
The Tech: They tried different AI methods to do this. They found that a newer type of AI (called SBERT, which understands context better than older methods) was much better at guessing the right topic.
Human Check: To keep the AI honest, humans regularly check the AI’s work and correct it. This helps the AI learn and get smarter over time, much like a student learning from a teacher’s corrections.

2. The Executive Summary Generator (Generative AI)

Once the comments are sorted, managers still don’t have time to read thousands of them. So, the team used Generative AI (like the technology behind ChatGPT) to write short summaries.

The Problem: At first, the AI was bad at this. It tended to focus only on the angry comments (because people complain more than they praise) and sometimes made things up (hallucinated) or contradicted itself.
The Solution: They changed the instructions given to the AI. Instead of asking for a general summary, they asked the AI to summarize comments by topic. For example: “Here are 50 comments about Performance. Please summarize the main issues.”
The Result: The AI now produces clean, structured reports. It also lists the specific comments it used to write the summary, so a human can quickly verify if the AI is telling the truth. This makes it easy for executives to see, at a glance, what users are thinking about specific features.

3. The "Angry Comment" Myth (Sentiment Analysis)

The most surprising part of the paper is about sentiment—whether a comment sounds happy (positive) or unhappy (negative).

Many managers assume that if a comment is negative, the user is unhappy with the product. This paper proves that is often wrong.

The Experiment: The researchers compared the "tone" of the comments (positive/negative) with actual satisfaction scores (like Net Promoter Score or Usability scores).
The Finding:
- Positive Comments: If a comment is positive, the user is almost always satisfied. This is a reliable signal.
- Negative Comments: If a comment is negative, the user might still be satisfied with the product overall.
Why? Think back to the restaurant. A customer might love the food and the service (high satisfaction) but leave a negative comment because the chair was wobbly. They are happy with the experience, but they are complaining about a specific flaw they hope will be fixed.
The Lesson: You cannot judge the overall quality of a product just by counting negative comments. A high number of negative comments doesn’t mean users hate the product; it often just means they care enough to point out specific problems. To know if users are truly satisfied, you must ask them directly with a rating scale, not just read their comments.

In Simple Terms

This paper shows that to understand user feedback at scale, you need:

AI to sort comments into categories (like "bugs" vs. "design").
AI to summarize those categories into easy-to-read reports.
Caution when reading tone: Don’t assume a negative comment means an unhappy customer. They might just be a happy customer who found a small thing to fix.

The goal is to combine the "big picture" numbers (ratings) with the "detailed stories" (comments) to get a complete, accurate view of how users feel.

Technical Summary: Integrating Multi-Label Classification and Generative AI for Scalable Analysis of User Feedback

Problem Statement
In competitive software markets, User Experience (UX) evaluation is critical for product success. Standard practice combines quantitative metrics (e.g., SUS, NPS, UX-Lite) with qualitative open-ended feedback. While quantitative metrics provide reliable indicators of overall satisfaction and usability, they often fail to explain why users perceive a product in a certain way. Conversely, analyzing large volumes of user comments manually is inefficient, costly, and inconsistent. Furthermore, there is a frequent mismatch between quantitative scores and comment sentiment; users may report high satisfaction while simultaneously describing specific negative issues. The paper addresses the challenge of efficiently processing, interpreting, and summarizing extensive volumes of user comments to support product teams and management.

Methodology
The authors present techniques developed within a long-term UX measurement project at SAP SE, focusing on three core components:

Multi-Label Comment Classification:
- Taxonomy Development: A supervised machine learning approach was employed using a pre-defined topic taxonomy. Labels were derived to be collectively exhaustive, consistently defined, and relevant to users. The initial codebook comprised ten distinct topic labels (e.g., Usability, Functionality, Error, Performance).
- Data Preparation: A training dataset was created by having two human coders independently annotate 2,756 user comments from 30 SAP Business Technology Platform (BTP) products. The dataset was expanded to 8,757 comments from 40 products through an incremental human-in-the-loop process.
- Embedding Generation: The authors compared two embedding methods:
  - fastText: A pre-trained shallow neural network with subword information, fine-tuned on SAP-specific lingo. Text preprocessing (stopword removal, lemmatization) was applied.
  - SBERT (Sentence-BERT): A Transformer-based model that generates contextual representations without extensive preprocessing.
- Model Architecture: Due to the multi-label nature of the data (20.86% of comments had multiple labels), the authors trained sequential binary classifiers for each label rather than a single multi-class model. After evaluating logistic regression, k-NN, SVM, and random forest, Extreme Gradient Boosting (XGBoost) was selected as the best performer. Hyperparameters were optimized using multi-label stratified $k$ -fold cross-validation to maximize the micro-averaged F1 score.
Generative AI (GenAI) Summarization:
- Initial attempts to generate general summaries or separate positive/negative summaries resulted in biased, inconsistent, or hallucinated outputs.
- The refined approach involves creating instructional prompts that task GenAI with producing summaries for specific predefined categories (from the classification model) that meet a minimum comment threshold.
- The prompt includes product-specific comments with IDs and classifications.
- A human-in-the-loop validation process ensures accuracy, selects verbatim snippets that support the summary, and maintains the distribution of positive and negative sentiments. Products with fewer than 20 comments are excluded from summarization to avoid overgeneralization.
Sentiment Analysis and Correlation Study:
- The authors analyzed the relationship between comment sentiment (Positive, Mixed, Negative, derived via Qualtrics) and quantitative UX metrics (NPS, Tutorial Quality Score, PSAT, UX-Lite) using two distinct surveys: one concerning tutorial quality and another concerning application usability.
- Statistical tests (Chi-squared, conditional probabilities, binomial significance tests) were used to assess whether sentiment could serve as a proxy for overall satisfaction.

Key Contributions

Automated Classification Pipeline: A robust, incremental multi-label classification system using SBERT embeddings and XGBoost, achieving high predictive performance (e.g., F1 score of 0.82 for "Licensing" and 0.76 for "Performance").
Structured GenAI Summarization: A validated workflow for generating concise, category-specific summaries of user feedback that mitigates GenAI hallucinations and bias by anchoring outputs to predefined topics and verbatim evidence.
Empirical Evidence on Sentiment Limitations: A detailed statistical analysis demonstrating that comment sentiment is not a reliable standalone indicator of overall product satisfaction.

Results

Classification Performance: The final multi-label model showed varying performance across categories. "Licensing" had the highest F1 score (0.82), while "Other" had the lowest (0.49), likely due to the heterogeneity of issues in that category. The use of SBERT improved the micro-averaged F1 score by 24.58% compared to fastText.
Sentiment vs. Satisfaction:
- Positive Sentiment: Strongly correlated with high satisfaction. In the application usability survey, 92% of respondents with positive comments reported being "Satisfied" or "Very Satisfied."
- Negative Sentiment: Poorly correlated with low satisfaction. In the tutorial survey, 47.24% of respondents who wrote negative comments were still classified as "Promoters" (high NPS). In the application usability survey, 22.15% of respondents with negative comments still reported being "Satisfied" or "Very Satisfied."
- Conclusion: Negative comments often describe specific, isolated issues (e.g., bugs, navigation difficulties) rather than overall dissatisfaction. Users may provide negative feedback on specific features while maintaining a positive overall impression of the product.

Significance and Claims
The paper claims that combining quantitative metrics with qualitatively enriched, AI-analyzed comments provides a complementary view of UX. Quantitative metrics offer robust indicators of overall quality, while comments provide contextual explanations and actionable insights.

The authors emphasize that sentiment analysis alone should not be used as a proxy for user satisfaction. A high ratio of negative sentiments does not necessarily indicate poor product quality; it may reflect users who are generally satisfied but are reporting specific, fixable issues. The paper advocates for explicit measurement of satisfaction in surveys and the use of AI-supported classification and summarization to efficiently interpret the nuanced context of user feedback, thereby avoiding the pitfall of equating negative comment sentiment with overall user dissatisfaction.

1. The Smart Sorting Hat (Topic Classification)

2. The Executive Summary Generator (Generative AI)

3. The "Angry Comment" Myth (Sentiment Analysis)

In Simple Terms

More like this