💬 NLP

From TF-IDF to Transformers: A Comparative and Ensemble Approach to Sentiment Classification

This paper evaluates various machine learning and NLP models for sentiment classification on the IMDb dataset, finding that RoBERTa achieves the highest accuracy at 93.02% while a soft voting ensemble of all models further enhances overall performance.

Original authors: Dip Biswas Shanto, Mitali Yadav, Prajwal Panth, Suresh Chandra Satapathy

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dip Biswas Shanto, Mitali Yadav, Prajwal Panth, Suresh Chandra Satapathy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how a movie audience feels just by reading their reviews. Some people write, "This was the best movie ever!" while others say, "I hated every minute of it." But sometimes, the truth is hidden in tricky phrases like, "It wasn't the worst movie I've ever seen," or sarcastic comments like, "Oh, great, another superhero movie that made no sense."

This paper is like a cooking competition where different chefs (computer models) try to taste these reviews and correctly guess if the diner is happy (positive) or unhappy (negative). The authors used a massive cookbook of 50,000 movie reviews from IMDb to train their chefs.

Here is how they tested their recipes:

1. The Old-School Chefs (Traditional Models)

First, they tried the "classic" methods. These models are like chefs who only look at a list of ingredients (words) and count how many times they appear.

The Method: They used a technique called TF-IDF, which is like a highlighter pen. It marks words that are important and unique to a specific review while ignoring common words like "the" or "and."
The Chefs: They used Naïve Bayes, Logistic Regression, SVM, and LightGBM.
The Result: These chefs were decent. They could spot obvious words like "good" or "bad." However, they sometimes got confused by complex sentences or sarcasm because they didn't really understand the story behind the words, just the ingredients.

2. The Smart Chefs (Deep Learning & Transformers)

Next, they brought in the "modern" chefs. These models are like chefs who not only read the ingredients but also understand the context and the flow of the recipe.

The Method: They used Transformers (specifically RoBERTa and DistilBERT). Think of these as super-smart readers that understand how words relate to each other in a sentence. They know that "not good" means something very different from just "good."
The Result: These chefs were much better. RoBERTa was the star of the show, getting it right about 93% of the time. It understood the nuances and metaphors that the old-school chefs missed. DistilBERT was a close second; it was like a lightweight version of the star chef—almost as smart but faster and easier to run.

3. The "All-Hands-On-Deck" Strategy (Ensemble Learning)

The authors realized that even the best single chef can make a mistake. So, they tried a Soft Voting Ensemble.

The Analogy: Imagine a panel of judges. Instead of one judge deciding the winner, you ask all the chefs (the old ones and the new ones) to vote. But instead of just shouting "Yes" or "No," they each give a percentage of confidence (e.g., "I'm 80% sure this is positive").
The Magic: The system takes all those percentages, averages them out, and makes a final decision.
The Result: This team approach worked even better than the single best chef. By combining their strengths, the group corrected each other's mistakes, especially on those tricky, sarcastic reviews. It was like having a safety net that caught errors no single model could see.

What Did They Learn?

The "Tricky" Reviews: The models still struggled with sarcasm (like saying "Great job" when they meant the opposite) and double negatives (like "not bad"). These were the hardest dishes to taste.
Transparency: The authors used a tool called SHAP to peek inside the black box. It's like putting a spotlight on the specific words that made the model change its mind. For example, it showed that the word "boring" was pulling the vote toward "negative," while "excellent" pulled it toward "positive."
The Winner: While the fancy RoBERTa model was the most accurate, the paper notes that the simpler, older models (like SVM) are still very useful if you don't have a super-computer to run the fancy ones.

The Bottom Line

The paper concludes that if you want the absolute best accuracy for understanding movie feelings, you should use the smartest Transformer model (RoBERTa) and combine it with a team voting strategy. However, if you need something quick and simple, the old-school methods are still surprisingly good at the basics.

In short: One smart chef is great, but a whole team of chefs working together is even better at guessing how people feel about a movie.

Technical Summary: From TF-IDF to Transformers: A Comparative and Ensemble Approach to Sentiment Classification

Problem Statement
Sentiment analysis, or opinion mining, aims to extract subjective opinions from text data, such as movie reviews, to classify them as positive or negative. While machine learning (ML) models are effective, they often struggle to accurately interpret context, metaphors, sarcasm, and complex negations because they rely heavily on statistical word representations rather than deep semantic understanding. This paper addresses the challenge of classifying IMDb movie reviews by comparing traditional statistical methods, deep learning architectures, and modern transformer-based models to determine the most efficient approach for resilient sentiment classification.

Methodology
The study utilizes the IMDb dataset, comprising 50,000 balanced movie reviews (25,000 positive and 25,000 negative). The methodology follows a structured pipeline:

Preprocessing: Raw text undergoes normalization, tokenization, and stemming/lemmatization. The dataset is split into training and testing sets.
Feature Engineering:
- Traditional Models: Text is converted to numerical features using TF-IDF vectorization, capped at the 10,000 most informative words with an n-gram range of 1–3. This prevents data leakage by fitting the vectorizer solely on the training corpus.
- Transformer Models: These utilize subword tokenization and contextual embeddings directly from pre-trained models, offering dynamic semantic representations compared to static TF-IDF features.
Model Training: Seven distinct classifiers were evaluated:
- Traditional ML: Logistic Regression, Multinomial Naïve Bayes, Linear SVM, and LightGBM.
- Deep Learning: Bidirectional LSTM (64 units, Adam optimizer).
- Transformers: DistilBERT and RoBERTa, fine-tuned using the Hugging Face framework with specific hyperparameters (e.g., sequence length of 128, weight decay of 0.01).
Ensemble Approach: A soft voting ensemble was constructed to combine the predicted probabilities of all seven models. Each model was assigned equal weight ( $w_i = 1/N$ ), and the final prediction was determined by the class with the highest averaged probability score.
Explainability: SHapley Additive exPlanations (SHAP) were employed to provide model transparency. For SVM, SHAP values corresponded to the product of TF-IDF weights and coefficients. For transformers, perturbation-based techniques were used to calculate token-level attributions.

Key Contributions and Results
The paper presents a comparative analysis of model performance based on accuracy, F1-score, and ROC-AUC:

Performance Hierarchy: Transformer-based models outperformed all other approaches. RoBERTa achieved the highest performance with 93.02% accuracy and an F1-score of 0.9302. DistilBERT followed closely (92.59% accuracy). Traditional models like SVM (89.61%) and Logistic Regression (88.91%) served as strong, interpretable baselines but lagged behind contextual models.
Ensemble Effectiveness: The soft voting ensemble, which aggregated predictions from all models, demonstrated that combining diverse architectures improves classification precision. The ensemble reduced variance and handled ambiguous cases (such as sarcasm and complex negations) better than individual models, yielding a 1–3% improvement across all metrics compared to the single best model (RoBERTa).
Error Analysis: The study identified specific failure modes, including complex negations (e.g., "not without its advantage") and sarcasm, which resulted in higher error rates (up to 18.9% for sarcasm). The ensemble approach notably reduced misclassification rates for sarcastic reviews by approximately 23%.
Ablation Insights: The study quantified the impact of specific components, finding that negation handling improved F1-scores by 3.2–4.1% for traditional models, and dynamic masking during RoBERTa fine-tuning contributed an additional 2.1% accuracy gain.

Significance and Claims
The paper concludes that while transformer-based models (specifically RoBERTa and DistilBERT) are superior for capturing contextual dependencies and subtle semantic nuances in complex sentiment analysis tasks, traditional methods remain valuable. SVM with TF-IDF features is highlighted as a computationally efficient and interpretable baseline, and DistilBERT is presented as a viable alternative when computational resources are limited.

The primary significance of this work lies in demonstrating that model ensembling effectively leverages the complementary strengths of diverse architectures (lexical, sequential, and contextual) to mitigate individual model biases and improve overall robustness. The authors assert that while transformers are best suited for complex tasks, a hybrid approach combining traditional efficiency with deep learning power offers a comprehensive solution for sentiment classification.

1. The Old-School Chefs (Traditional Models)

2. The Smart Chefs (Deep Learning & Transformers)

3. The "All-Hands-On-Deck" Strategy (Ensemble Learning)

What Did They Learn?

The Bottom Line

More like this