💬 NLP

Improving Customer Service with Automatic Topic Detection in User Emails

This study presents a production-ready, language-agnostic NLP pipeline using BERTopic that achieves high accuracy (0.96 F1 score) and rapid processing (0.041 seconds per email) to automatically classify and label Serbian customer service emails, significantly improving operational efficiency for Telekom Srbija.

Original authors: Bojana Bašaragin, Darija Medvecki, Gorana Gojić, Milena Oparnica, Dragiša Mišković

Published 2026-02-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bojana Bašaragin, Darija Medvecki, Gorana Gojić, Milena Oparnica, Dragiša Mišković

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine Telekom Srbija, a giant telecommunications company, as a massive, bustling post office. Every day, thousands of people drop off letters (emails) with complaints, questions, and requests. Before this new system, a team of human workers had to read every single letter, figure out what it was about, and then sort it into the right pile. It was slow, tiring, and easy to get overwhelmed.

This paper describes how the company built a super-smart robot librarian to do the sorting for them. Here is how it works, broken down into simple steps:

1. The Problem: A Mountain of Unsorted Mail

The company was drowning in emails. Some were about broken internet, others about billing errors, and some were just spam or internal notes. The humans needed a way to instantly know, "Oh, this letter is about a bill," or "This one is about a broken router," so they could send it to the right expert immediately.

2. The Solution: The "Topic Detective" (BERTopic)

Instead of teaching the computer to memorize every possible sentence (which is impossible), they used a tool called BERTopic. Think of BERTopic as a detective that looks for patterns in the "vibe" of the words, rather than just matching keywords.

The Magic Brain: The detective uses a "multilingual brain" (a pre-trained AI model) that understands Serbian, English, and many other languages. It doesn't need to be taught from scratch; it just needs to be shown a few examples of what the company cares about.
The Grouping Game: The detective reads thousands of emails and says, "Hey, these 500 emails all talk about 'money' and 'invoices,' so let's put them in the Bill pile. And these 300 emails talk about 'slow internet' and 'connection,' so they go in the Technical Issue pile."

3. Cleaning the Mailbox (Preprocessing)

Before the detective can read the letters, the team had to clean up the mailbox. They built a filter that:

Translates scripts: Converts old-school Cyrillic letters into modern Latin letters so the robot can read them all the same way.
Removes the noise: Throws away emojis, random numbers, and polite sign-offs like "Best regards" because those don't tell the robot what the problem is.
Cuts the fluff: If an email is too short (like just "Hello"), the robot ignores it because it has no clues. If it's too long, the robot focuses on the first part, which usually contains the main problem.

4. The "72 to 12" Trick

When the robot first started sorting, it was too good. It created 72 different piles (topics). That was too many for the human workers to manage.

The Analogy: Imagine the robot sorted mail into 72 tiny drawers. The humans said, "We only have 12 big bins!"
The Fix: The humans manually looked at the 72 tiny drawers and said, "Okay, these 5 drawers are all about 'Bills,' so let's glue them together into one big 'Bills' bin." They did this until they had exactly 12 clear categories that matched the company's needs.

5. The Results: Speed and Accuracy

The new system is incredibly fast and accurate:

Speed: It sorts an email in 0.04 seconds. That's faster than you can blink. It can process thousands of emails in the time it takes to make a cup of coffee.
Accuracy: It gets it right 96% of the time. If a human were sorting 100 emails, they might make a few mistakes due to fatigue. The robot rarely does.

6. How It Helps in Real Life

Now, the system works in the background like a silent traffic cop.

An email arrives.
The robot instantly tags it: "Topic: Billing Issue."
The human customer service agents use a special app to filter their inbox. They can click "Billing," and poof, all the billing emails appear instantly.
They can also spot trends. If suddenly 500 emails come in about "Router Crashes," the team knows immediately that there is a network outage and can fix it before customers get angry.

Why This Matters

The best part? This robot isn't just for Serbian. Because it uses a multilingual brain, it could easily be taught to sort emails in Turkish, Russian, or any other language, even languages that are "hard" for computers to learn. It turns a chaotic mountain of letters into an organized, easy-to-read library, letting humans focus on solving problems instead of just sorting paper.

1. Problem Statement

Telekom Srbija, a major telecommunications provider, faces challenges in managing high volumes of incoming customer emails. Manual processing is inefficient, leading to delays in addressing customer concerns. The specific challenges include:

Volume and Complexity: Handling large datasets with semantic and syntactic ambiguity.
Language Constraints: The need to process Serbian, a low-resource and morphologically rich language, where pre-trained NLP tools and datasets are scarce.
Operational Efficiency: The need to route emails to the correct departments quickly to improve customer experience and reduce costs.

The goal is to develop an automated Natural Language Processing (NLP) pipeline to detect topics in user emails, label them, and route them efficiently without requiring extensive manual labeling or language-specific resources.

2. Methodology

The authors developed an unsupervised NLP pipeline centered around BERTopic, a modular framework for topic modeling. The process consists of three main stages:

A. Data Preparation

Dataset: Approximately 90,000 anonymized emails (50k individual, 40k business users) from Telekom Srbija.
Preprocessing Steps:
1. Script Standardization: Transliteration of Cyrillic to Latin script to ensure consistent embeddings.
2. Cleaning: Removal of punctuation, numbers, special characters, and emojis.
3. Length Filtering: Removal of emails with $\le$ 3 words (uninformative) and emails exceeding 128 tokens (BERT model limit).
4. Noise Reduction: Removal of duplicates, closing phrases, signatures, and anonymization placeholders.
5. Spam/Filtering: Exclusion of non-Serbian emails (mostly spam) and automated replies using langid and keyword filtering.
6. Lemmatization: Skipped intentionally, as standard lemmatizers fail on company-specific terminology, and prior research indicated BERTopic performs well on unlemmatized Serbian text.

B. Model Training (Unsupervised)

The pipeline utilizes BERTopic with the following configuration:

Embedding: paraphrase-multilingual-mpnet-base-v2 (multilingual sentence transformer) to handle Serbian within a multilingual context.
Dimensionality Reduction: UMAP (default settings) to reduce vector dimensionality.
Clustering: HDBSCAN with min_topic_size set to 100. This was chosen to balance the creation of too many small clusters vs. too few large, mixed clusters.
Topic Representation: CountVectorizer and c-TF-IDF. Stop words were expanded iteratively based on manual review of initial keywords.
Outlier Handling: The reduce_outliers method using c-TF-IDF strategy was applied, reducing outliers from ~47,000 to ~470 documents.

C. Post-Processing and Topic Reduction

Initial Output: The model initially generated 72 distinct topics.
Manual Aggregation: Since 72 topics were too granular for operational use, the team manually analyzed keywords and representative emails to map the 72 topics into 12 derived, business-relevant topics (e.g., "Bills and Invoices," "General Problems").
Implementation Strategy: Instead of retraining the model to force 12 clusters (which failed to produce the desired semantic separation), the team used a labeling strategy. They assigned the 12 custom labels to the existing 72 clusters and the outlier group, creating a hierarchical mapping.

3. Key Contributions

Language-Agnostic Pipeline for Low-Resource Languages: Demonstrated that a multilingual embedding model combined with BERTopic can effectively model topics in Serbian without extensive lemmatization or domain-specific pre-training.
Hybrid Unsupervised-Supervised Approach: Developed a novel workflow where unsupervised clustering is used to discover semantic structures, followed by a manual, rule-based aggregation step to align with business logic. This avoids the pitfalls of forcing a specific number of clusters during training.
Production-Ready System: Successfully deployed the system in a live production environment, integrating it with the company's database and a custom customer service application.
Handling Morphologically Rich Text: Proved effective handling of Serbian text (rich morphology) by avoiding traditional lemmatization and relying on contextual embeddings.

4. Results

The system was evaluated on a test set of 100 manually reconstructed emails (restoring anonymized entities).

Performance Metrics:
- Weighted Average F1 Score: 0.96
- Accuracy: 0.96
- Weighted Average Precision: 0.9645
- Weighted Average Recall: 0.9600
Processing Speed:
- Average processing time: 0.041 seconds per email.
- The system scales linearly with batch sizes (tested up to 10,000 emails), making it suitable for real-time or near-real-time batch processing.
Error Analysis:
- Errors occurred primarily when emails contained keywords from multiple topics, causing the model to prioritize a secondary topic.
- The model successfully identified the dominant topic in 96% of cases.
- Attempts to detect secondary topics (using probability scores) were found to be insufficiently accurate for production use.

5. Significance and Future Work

Operational Impact: The system is currently live at Telekom Srbija, enabling customer service agents to filter emails by category, identify dissatisfied customers proactively, and generate monthly trend reports.
Scalability: The methodology is transferable to other low-resource, morphologically rich languages with minimal adaptation.
Future Directions:
- Multi-Label Classification: Developing mechanisms to assign multiple topics to a single email (as customers often have multiple issues).
- Long-Document Processing: Overcoming the 128-token limit to process full-length emails without truncation.
- Novelty Detection: Implementing a mechanism to flag emerging topics that do not fit existing clusters.
- Confidence Scoring: Adding confidence scores to predictions to flag low-confidence emails for human review.

In conclusion, this study presents a highly effective, production-grade solution for automating customer service workflows in a challenging linguistic environment, achieving high accuracy and speed while remaining adaptable to future business needs.