💬 NLP

Beyond the Star Rating: A Scalable Framework for Aspect-Based Sentiment Analysis Using LLMs and Text Classification

This study proposes a scalable hybrid framework that leverages large language models for aspect identification and traditional machine learning for sentiment classification to effectively analyze millions of customer reviews, demonstrating that this approach can significantly explain variance in overall restaurant ratings across diverse contexts.

Original authors: Vishal Patil, Shree Vaishnavi Bacha, Revanth Yamani, Yidan Sun, Mayank Kejriwal

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vishal Patil, Shree Vaishnavi Bacha, Revanth Yamani, Yidan Sun, Mayank Kejriwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you own a massive library containing 4.7 million books. These aren't just any books; they are reviews written by customers about restaurants over the last 17 years. You want to know exactly why some restaurants get five stars and others get one star. Is it the food? The service? The wait time? The price?

Reading every single book to find the answers would take a human lifetime. That's where this paper comes in. The authors built a super-smart, automated librarian that can read all those millions of reviews in a flash and tell you exactly what people are talking about and how they feel.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The "Needle in a Haystack"

Customer reviews are messy. A single review might say, "The food was amazing, but the waiter was rude, and the wait was too long."

The Challenge: Computers are usually bad at understanding that "food" is good, "waiter" is bad, and "wait" is bad all in the same sentence.
The Old Way: Researchers used to try to find patterns by counting words (like "good" or "bad"), but that's like trying to understand a movie by just counting how many times the word "run" appears. It misses the context.

2. The Solution: The "Hybrid Librarian" Team

The authors realized that using a super-powerful AI (like ChatGPT) to read every single one of the 4.7 million reviews would be too expensive and slow. It's like hiring a team of 100 PhD professors to read every book in the library; it would cost a fortune.

Instead, they created a two-part team:

Part A: The Expert Architect (The LLM/ChatGPT)
They hired the "Professor" (ChatGPT) to read a small sample of 600 reviews. The Professor's job wasn't to grade every book, but to draw up the blueprint. The Professor said: "Okay, when people talk about restaurants, they usually care about six things: Food Quality, Service, Ambiance, Wait Time, Price, and Menu Variety."
- Analogy: Think of the Professor as the architect who designs the filing system.
Part B: The Fast Sorters (Traditional Machine Learning)
Once the blueprint was ready, they trained a team of fast, cheap, local computers (traditional machine learning models) to do the heavy lifting. These computers learned from a smaller set of human-labeled reviews (5,000 reviews) and then went on to sort the remaining 4.7 million reviews based on the Professor's blueprint.
- Analogy: These are the efficient interns who can sort millions of files in seconds once they know the rules.

3. The Two Ways of Sorting (The "One-Stage" vs. "Two-Stage" Debate)

The researchers tried two different ways to organize the files:

The "One-Stage" Method: The computer reads a review and immediately guesses the sentiment for all six categories. If the review doesn't mention "Wait Time," the computer just guesses "Neutral" (no opinion).
- Result: This was the winner. It was faster and more accurate overall.
The "Two-Stage" Method: The computer first asks, "Does this review even mention Wait Time?" If yes, it moves to step two to guess the sentiment. If no, it skips it.
- Result: This was more careful about finding negative comments, but it got confused more often and was slower.

The Verdict: The "One-Stage" method was the best all-rounder.

4. The Big Discovery: What Actually Makes a Restaurant a Star?

After sorting all 4.7 million reviews, they ran a statistical test to see which of the six categories actually explained why a restaurant got a high rating.

Here is what they found (the "Secret Sauce"):

Food Quality is King: This was the biggest factor. If the food is good, the rating goes up. It's the most important thing.
Service is the Queen: Good service is the second most important thing.
Menu Variety: Having lots of options helps.
The Wait Time Surprise: You might think a long wait makes people angry. But the data showed that longer waits actually correlated with higher ratings!
- Why? It's the "Social Proof" effect. If a place is crowded and you have to wait, people think, "Wow, this must be the best place in town!"
The Ambiance Twist: Surprisingly, the "vibe" or decor had a negative relationship with ratings in their model.
- Why? The authors suspect this is because "ambiance" is hard to define. One person's "cozy" is another person's "cramped." Because the data was messy here, the computer got confused, leading to a weird result.
Price: Surprisingly, price didn't matter much. People didn't seem to care if it was expensive or cheap as long as the food and service were good.

5. Why This Matters

This paper proves that you don't need to spend millions of dollars to analyze customer feedback. By using a smart AI just to set the rules, and then using cheaper, faster computers to do the actual work, businesses can understand millions of customers instantly.

In a nutshell:
They built a system that acts like a super-fast translator. It takes the chaotic, messy words of millions of customers and turns them into a clear, simple report card for restaurant owners, showing them exactly what to fix to get more stars.

Food? Keep it delicious.
Service? Be friendly.
Wait? Don't worry too much; a long line might actually mean you're popular!

1. Problem Statement

Customer reviews are a critical data source for businesses, particularly in the hospitality industry, yet analyzing millions of unstructured reviews remains a significant challenge. Traditional methods face two main hurdles:

Scalability and Cost: Large Language Models (LLMs) like ChatGPT show promise for natural language understanding but are computationally expensive and impractical for processing massive datasets (e.g., millions of reviews) directly.
Granularity: Standard star ratings (1–5) are aggregate metrics that fail to capture specific drivers of satisfaction (e.g., food quality vs. service). Existing Aspect-Based Sentiment Analysis (ABSA) methods often struggle to adapt across different domains and geographies or rely on manual thematic analysis, which is intractable at scale.

The core problem is how to automate the identification of specific dining aspects and their associated sentiments across a massive corpus (4.7 million reviews) in a cost-effective, scalable manner while maintaining high accuracy.

2. Methodology

The authors propose a hybrid framework that leverages LLMs for aspect discovery and traditional machine learning (ML) for scalable sentiment classification.

A. Data Collection

Source: Yelp Open Dataset (2005–2022).
Scope: 4.7 million reviews from 52,286 distinct restaurants across 17 U.S. states and one Canadian province.
Preprocessing: Merged review and business data, filtering strictly for restaurant categories.

B. Aspect Discovery (LLM Phase)

Sampling: 600 reviews from 30 California restaurants were analyzed.
Tool: ChatGPT-4o and ChatGPT-4o-mini were prompted to extract key dining aspects.
Validation: The LLM-derived aspects were compared against Latent Dirichlet Allocation (LDA) topic modeling. While LDA identified general themes, it lacked specific aspect granularity.
Result: Six key aspects were selected for the study: Service, Food Quality, Ambiance, Wait Time, Price, and Menu Variety.

C. Sentiment Labeling & Model Training

Human Annotation: A subset of 5,000 reviews was manually annotated by five human raters for the six aspects. Labels included Positive (+1), Neutral (0), Negative (-1), and Irrelevant (NA).
Inter-Annotator Reliability: High agreement was observed using Fleiss' Kappa (e.g., Price $\kappa=0.943$ , Service $\kappa=0.727$ ).
Vectorization: Two methods were tested: TF-IDF and fastText (trained on the full 4.7M corpus).
Classification Architectures: Two approaches were evaluated:
1. One-Stage Classifier: Treats "irrelevant" reviews as neutral (3-class: Neg, Neu, Pos).
2. Two-Stage Classifier: Stage 1 determines relevance (Binary); Stage 2 predicts sentiment only for relevant reviews (3-class).
Model Selection: Four algorithms (Naive Bayes, Logistic Regression, Random Forest, SVM) were tested. Logistic Regression with fastText was selected as the optimal model due to its balance of accuracy, robustness, and computational efficiency compared to SVM (which was too slow and had poor recall for certain classes).

D. Regression Analysis

Aggregation: Review-level sentiment predictions were aggregated to the restaurant level (averages per aspect).
Goal: To determine if aspect-based sentiments explain the variance in overall restaurant ratings.
Models: Four linear regression specifications were fitted:
1. Base (no controls).
2. Cuisine-controlled.
3. State-controlled.
4. Fully controlled (Cuisine + State).

3. Key Contributions

Scalable Hybrid Architecture: Demonstrated a practical framework combining LLMs (for zero-shot aspect discovery) with lightweight, locally deployable ML classifiers (for massive-scale sentiment labeling).
Architectural Comparison: Provided a rigorous comparison between One-Stage and Two-Stage classifiers. Surprisingly, the One-Stage classifier outperformed the Two-Stage approach across all aspects (validated via McNemar's test), likely due to the Two-Stage model's difficulty in correctly identifying "neutral/irrelevant" instances in the second stage.
Quantitative Impact Analysis: Moved beyond qualitative descriptions to statistically quantify how specific aspects drive overall ratings using regression analysis on a massive dataset.

4. Key Results

A. Classification Performance

Best Model: Logistic Regression with fastText achieved the highest average accuracy (0.766).
One-Stage vs. Two-Stage: The One-Stage classifier significantly outperformed the Two-Stage classifier in overall accuracy and F1-scores across all aspects. For example, on the "Food Quality" aspect, the One-Stage classifier had an accuracy of 0.78, while the Two-Stage Stage 2 had 0.76.
McNemar Test: The performance difference was statistically significant ( $p < 0.001$ ) for all aspects, confirming the One-Stage approach is superior for this specific task.

B. Regression Analysis (Impact on Ratings)

The regression models showed strong goodness-of-fit ( $R^2$ ranging from 0.814 to 0.819), indicating that aspect sentiments explain a vast majority of the variance in overall ratings.

Food Quality: The strongest predictor ( $\beta \approx 1.58$ , $p < 0.001$ ).
Service: The second strongest predictor ( $\beta \approx 0.75$ , $p < 0.001$ ).
Menu Variety: Significant positive impact ( $\beta \approx 0.67$ ).
Wait Time: Surprisingly, showed a positive correlation ( $\beta \approx 0.22$ ), suggesting longer waits may signal high demand/popularity rather than dissatisfaction.
Ambiance: Showed a negative correlation ( $\beta \approx -0.27$ ), a counter-intuitive finding the authors attribute to potential labeling noise or specific context in the data.
Price: Not statistically significant.
Controls: Italian and Chinese cuisines were associated with higher ratings compared to American cuisine. Regional variations were also significant (e.g., Montana showed a large positive effect, though likely due to small sample size).

5. Significance and Implications

Practical Framework: The study offers a blueprint for researchers and industry practitioners to automate ABSA at a scale previously impossible without prohibitive costs. It proves that LLMs do not need to process every single review; they can be used to define the schema, while cheaper ML models handle the volume.
Business Intelligence: The findings validate that specific aspects (especially food and service) are the primary drivers of revenue and reputation, allowing businesses to prioritize improvements based on data rather than intuition.
Methodological Insight: The unexpected negative correlation for ambiance and positive correlation for wait time highlights the complexity of consumer psychology and the value of large-scale quantitative analysis over small-sample qualitative studies.
Future Applications: The framework is adaptable to other service sectors (retail, healthcare) and platforms (Google Reviews, TripAdvisor), and can be extended to track temporal trends in customer preferences.

Limitations: The study relied on a limited initial sample (600 reviews) for aspect definition and focused on only six aspects. The negative finding for ambiance suggests a need for further investigation into labeling consistency for subjective attributes.