💬 NLP

Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Model

This study demonstrates that for Turkish sentiment analysis on e-commerce reviews, supervised fine-tuning of models like BERTurk still outperforms zero-shot prompting with large language models, particularly in accurately classifying the challenging neutral category.

Original authors: Sercan Karakaş, Yusuf Şimşek

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sercan Karakaş, Yusuf Şimşek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to read customer reviews for an online store in Turkey and decide if the customer is happy (positive), upset (negative), or just meh (neutral).

This paper asks a big question: Do we still need to spend time and money "training" (fine-tuning) a specific computer model for this job, or can we just ask a super-smart, general-purpose AI (a Large Language Model) to do it on the spot with a simple question (prompting)?

Here is the breakdown of their findings using some everyday analogies:

1. The Contestants

The researchers set up a race between three types of "students":

The General Geniuses (Prompted LLMs): These are like brilliant, well-read students who know everything about the world but have never taken a specific test on Turkish e-commerce reviews. You just ask them, "Is this review good or bad?" and they give an answer.
The Specialized Interns (Fine-Tuned Models): These are students who took the exact same test many times before. They studied the specific quirks of Turkish online shopping reviews.
The Old School Tutors (Classical Machine Learning): These are the traditional methods that rely on counting words and patterns without deep "understanding."

2. The Test: The "Meh" Problem

The test wasn't just "Good vs. Bad." It included a third, tricky category: "Neutral."

Think of a neutral review like a customer saying, "The shirt fits okay, the fabric is standard, and it arrived on time." It's not a rave, but it's not a complaint. It's the middle ground.

The Results:

The Specialized Interns won. The models that were specifically trained (fine-tuned) on Turkish data performed the best overall. They got the highest score (about 83.7% accuracy).
The General Geniuses struggled with the middle ground. While the super-smart AI models were decent at spotting clear "Happy" or "Sad" reviews, they fell apart when faced with the "Meh" reviews.

3. The Main Issue: The "Polarizing" Trap

The paper found that the big AI models have a bad habit: They hate the middle.

When the AI models saw a "Neutral" review, they often panicked and forced it into one of the two extreme boxes.

The Metaphor: Imagine a scale that only has "Heavy" and "Light." If you put a medium-weight rock on it, the scale doesn't know what to do, so it randomly guesses "Heavy" or "Light."
The Reality: The AI models often took a neutral Turkish review and labeled it as "Positive" (Happy) just to be safe. For example, one model (Llama 3.1) took 70% of the actual "Neutral" reviews and incorrectly called them "Positive."

4. The "Binary" Trick

The researchers tried a trick: What if we removed all the "Neutral" reviews and just asked the AI to choose between "Happy" and "Sad"?

Suddenly, the General Geniuses got much better. Their scores jumped up significantly.
The Specialized Interns improved too, but not as much. This tells us that the Interns were already good at understanding the whole picture (including the middle), while the Geniuses were only good at spotting the extremes.

5. The Bottom Line

The paper concludes that for Turkish sentiment analysis, you still need to do the hard work of fine-tuning.

Why? Because real-life reviews are messy. They aren't just black and white; they are full of gray areas.
The Lesson: If you just ask a general AI to guess, it will likely ignore the "gray areas" and force everything into "Good" or "Bad." To get a truly accurate reading of customer feelings in Turkey, you need a model that has been specifically trained to recognize and respect that "Meh" category.

In short: The super-smart AI is great at the basics, but for the nuanced, real-world task of understanding Turkish customer reviews (especially the boring, neutral ones), a specialized, trained model is still the champion.

Technical Summary: Do We Still Need Fine Tuning? Turkish Sentiment Analysis in the Era of Large Language Models

Problem Statement
This study investigates whether supervised fine-tuning remains necessary for Turkish sentiment analysis in the era of Large Language Models (LLMs). While instruction-following LLMs offer zero-shot and few-shot capabilities, their reliability on fine-grained sentiment tasks—specifically those requiring a three-class distinction (positive, negative, and neutral)—remains uncertain. The research addresses a gap in Turkish NLP resources, where prior work has established the efficacy of supervised and BERT-based models, but the performance of prompted LLMs on realistic, multi-class e-commerce reviews has not been systematically compared. A central challenge identified is the "neutral" class, which often involves factual descriptions, weak evaluativity, or mixed affect, making it harder to classify than binary polarities.

Methodology
The authors conducted a controlled comparative study using a dataset of 6,367 Turkish e-commerce reviews, annotated into three sentiment categories: negative (0), neutral (1), and positive (2). The dataset was split into 80% training and 20% testing (1,274 instances).

The experimental setup evaluated three distinct approaches on the same test partition:

Classical Machine Learning: Logistic Regression, Support Vector Machines (SVM), Random Forest, and Naive Bayes, utilizing TF-IDF features (max 5,000 features).
Supervised Fine-Tuned Models: Pretrained transformer models including BERTurk (32k and 128k Cased) and Turkish ELECTRA, fine-tuned with a learning rate of $2\times10^{-5}$ , 5 epochs, and a batch size of 32.
Prompted Large Language Models: Seven general-purpose LLMs (Gemma2:9B, Gemma3:27B, GPT-OSS:20B, Llama 3.1:8B, Magibu:11B, and Qwen3:32B) evaluated in a zero-shot setting using a standardized prompt and fixed decoding parameters (temperature 0.1, top_p 1).

Performance was measured using accuracy, precision, recall, and F1-score. The study also included a binary re-analysis (excluding neutral instances) to isolate the difficulty of the neutral class.

Key Results

Overall Performance: Fine-tuned BERTurk models achieved the highest performance. BERTurk Cased 128k reached an accuracy of 0.837 and a weighted F1 of 0.834. This significantly outperformed the best prompted LLM, Qwen3:32B (0.773 accuracy), by approximately 6.4 percentage points, representing a ~28% reduction in errors compared to the strongest LLM baseline.
The Neutral Class Challenge: The primary source of error was the neutral class (22.6% of the test set). While BERTurk models achieved neutral recall scores of roughly 0.60, most LLMs struggled significantly (e.g., Qwen3:32B at 0.312, Llama 3.1:8B at 0.160).
Polarization Tendency: Many LLMs exhibited a strong bias toward collapsing the neutral class into polarized categories. For instance, Llama 3.1:8B mapped 70.1% of ground-truth neutral instances to the positive class, and Magibu:11B mapped 60.8% to positive.
Binary vs. Ternary Performance: When neutral instances were removed to create a binary task, LLM performance improved dramatically (e.g., Qwen3:32B rose to 0.908 accuracy). In contrast, the fine-tuned BERTurk model showed a more modest gain (0.837 to 0.904). This suggests that fine-tuned models inherently encode the ternary decision structure, whereas LLMs rely more heavily on coarse polarity cues that become sufficient only when the neutral boundary is removed.

Key Contributions

Empirical Benchmark: The study provides a rigorous comparison of classical, fine-tuned, and prompted approaches specifically for Turkish three-class sentiment analysis, highlighting that supervised fine-tuning remains superior in this specific domain.
Diagnostic Value of the Neutral Class: The paper demonstrates that the neutral class serves as a critical "stress test" for model competence. High performance on binary tasks can mask a model's inability to distinguish nuanced, non-polarized sentiment.
Methodological Insight: The authors argue that evaluations relying solely on positive-negative distinctions may overestimate the competence of prompted LLMs by obscuring failures on the most interpretively demanding parts of the label space.

Significance and Claims
The paper claims that for realistic Turkish sentiment classification involving a neutral category, supervised fine-tuning is currently more effective than zero-shot prompting with general-purpose LLMs. The authors posit that the advantage of fine-tuned models lies not merely in raw accuracy but in their ability to align with the specific annotation scheme and preserve the full ternary structure of the task.

The findings suggest that while LLMs are competitive in coarse polarity detection, they degrade substantially in three-class settings by collapsing neutral reviews into polarized categories. Consequently, the study concludes that in realistic settings where annotation granularity includes a neutral class, specialized fine-tuning remains necessary to achieve robust performance, and that the inclusion of the neutral class is crucial for a valid evaluation of sentiment analysis systems.