💬 NLP

Topic Modeling with Fine-tuning LLMs and Bag of Sentences

This paper introduces FT-Topic, an unsupervised fine-tuning approach that automatically constructs training datasets from sentence groups to enhance LLM-based topic modeling, demonstrating its effectiveness through the novel, state-of-the-art SenClu method.

Original authors: Johannes Schneider

Published 2026-02-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Johannes Schneider

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library filled with millions of books, but no one has ever organized them. There are no shelves, no labels, and no catalog. If you walked in, you'd see a chaotic mess of stories, news, and opinions all jumbled together.

Topic Modeling is the art of automatically sorting this library into neat categories (like "Sports," "Politics," or "Cooking") so you can find what you're looking for.

For a long time, computers tried to do this by counting words (like counting how many times "goal" or "ball" appears). But this is like trying to understand a movie just by counting how many times the word "the" appears. It misses the meaning.

Recently, we got "Super-Intelligent Librarians" called Large Language Models (LLMs). These are AI models that understand context and nuance. They are great at reading a sentence and knowing what it's about. But the author of this paper noticed a problem: We were using these super-intelligent librarians "out of the box," without teaching them how to sort our specific library.

Here is the simple breakdown of what this paper does:

1. The Problem: The "Out-of-the-Box" Librarian

Imagine you hire a brilliant librarian who has read every book in the world. You ask them to sort your library. They are smart, but they don't know your rules. They might sort a book about "Apple the fruit" and "Apple the tech company" into the same pile because they both contain the word "Apple."

In the past, researchers just used these pre-trained AI models without teaching them specifically for sorting topics. The paper argues: "We need to fine-tune them!" But there's a catch: To teach an AI, you usually need a teacher with a stack of labeled books (e.g., "This is Sports," "This is Politics"). Creating those labels is expensive and takes forever.

2. The Solution: "FT-Topic" (The Self-Teaching Librarian)

The author, Johannes Schneider, invented a clever trick called FT-Topic. It's like giving the librarian a set of rules so they can teach themselves without needing a human teacher.

Here is the magic trick:

The "Bag of Sentences" Rule: Instead of looking at single words (like "dog") or entire books (which might have 50 different topics), the AI looks at small groups of sentences (like a paragraph).
The Logic: The AI assumes that sentences right next to each other in a book are usually talking about the same thing. Sentences from a completely different book are likely talking about something else.
The Self-Teaching Loop:
1. The AI grabs a group of sentences.
2. It guesses: "The sentences next to this one are similar (Same Topic). Sentences from a random other book are different (Different Topic)."
3. It creates a training game: "Make these two groups look alike, and make these two groups look different."
4. The Quality Control: Sometimes the AI gets it wrong (e.g., a book switches topics halfway through). The paper adds a "smart filter" that checks the AI's own guesses. If the AI thinks two sentences are similar but they actually look very different, it throws that example out. This cleans up the training data automatically.

By the end of this process, the AI has a custom "brain" specifically tuned to understand the topics in your documents.

3. The New Sorting Machine: "SenClu"

Once the AI is trained, the paper introduces a new sorting machine called SenClu.

Hard Assignments: Imagine a bookshelf where a book can only sit in one slot. Some old methods tried to say, "This book is 40% Sports and 60% Politics." SenClu says, "No, this book belongs in the Sports section." This is faster and easier for humans to understand.
The "Annealing" Trick: Imagine you are trying to organize a messy room. If you just start throwing things into piles, you might get stuck in a bad arrangement. The paper uses a technique called Simulated Annealing (like cooling molten metal). It starts by being a little chaotic (allowing mistakes) and slowly gets stricter. This helps the AI escape "bad arrangements" and find the best possible organization.
User Control: You can tell the AI, "I want every document to have about 2 topics," or "I want them to have 5." It listens to you.

4. Why is this better? (The Results)

The paper tested this against the old ways (like LDA) and other new AI methods.

Speed: It's much faster than the heavy, complex AI methods that take hours to run. It takes minutes.
Quality: The topics it finds make more sense to humans. Instead of a list of random words, you get clear themes.
Flexibility: It can find multiple topics in a single document, which older methods often missed.

The Big Picture Analogy

Think of Topic Modeling as trying to sort a giant pile of mixed-up LEGO bricks.

Old Methods (LDA): Counted the colors of the bricks. If a brick is red, it goes in the "Red" pile. But a red brick could be a car, a house, or a fire truck. The piles get messy.
Standard AI (BERT out-of-the-box): A smart robot looks at the bricks. It knows a red brick is often a fire truck. But it hasn't been told what your specific collection looks like, so it still makes mistakes.
This Paper (FT-Topic + SenClu): We give the robot a set of rules to look at clusters of bricks (a small section of a model) rather than single bricks. We let the robot practice sorting these clusters against each other, cleaning up its own mistakes as it goes. Finally, it builds a sorting system that is fast, accurate, and understands your specific collection perfectly.

In short: The paper teaches AI how to teach itself to organize text, making it faster, smarter, and more useful for real-world problems without needing expensive human teachers.

1. Problem Statement

Traditional topic modeling methods, such as Latent Dirichlet Allocation (LDA), rely on the "Bag of Words" (BoW) assumption and word frequency. These approaches suffer from several limitations:

Semantic Fragmentation: They often produce fragmented topics because they treat words as independent units, ignoring context.
Coherence Issues: Optimization objectives like perplexity often correlate poorly with human-perceived topic coherence (measured by PMI).
Static Embeddings: While pre-trained Large Language Models (LLMs) like BERT offer superior contextual embeddings, they are typically used "out-of-the-box." These models are not optimized specifically for the similarity metrics required in topic modeling (e.g., predicting missing words vs. identifying topic similarity).
Data Scarcity for Fine-tuning: Fine-tuning LLMs for specific tasks usually requires large, labeled datasets, which are expensive and scarce to produce.

The paper addresses the challenge of how to unsupervisedly fine-tune LLM encoders to improve topic modeling performance without relying on human-labeled data, while also proposing a new modeling unit that balances the granularity of words and entire documents.

2. Methodology

The paper introduces two core components: FT-Topic (a fine-tuning framework) and SenClu (a novel topic modeling algorithm).

A. The "Bag of Sentences" (BoS) Unit

Instead of using individual words (BoW) or entire documents (Document Clustering), the authors propose using groups of consecutive sentences as the elementary unit of analysis.

Rationale: A sequence of a few sentences usually contains sufficient semantic information to belong to a single topic, avoiding the unnatural topic switching that occurs word-by-word in BoW models.
Structure: A document is treated as a sequence of disjoint sentence groups ( $g_i$ ).

B. FT-Topic: Unsupervised Fine-Tuning

To overcome the lack of labeled data, the authors propose an automatic dataset construction method to fine-tune sentence encoders:

Heuristic Data Generation:
- Positive Pairs: Adjacent sentence groups within the same document are assumed to share the same topic.
- Negative Pairs: Random sentence groups from different documents are assumed to belong to different topics.
Quality Improvement (Data Cleaning): The initial heuristic assumption is imperfect (adjacent sentences might switch topics; distant documents might share topics). The authors introduce a filtering step:
- They use a pre-trained (non-fine-tuned) model to compute similarity scores for the generated pairs.
- Filtering Logic: Pairs assumed to be "positive" (same topic) but showing low similarity are removed. Triplets where the difference between positive and negative similarity is small (indicating ambiguity) are also removed.
Fine-Tuning: The cleaned dataset is used to fine-tune a sentence encoder (e.g., Sentence-BERT) using Triplet Loss. This optimizes the embedding space so that sentences from the same topic are closer together than those from different topics.

C. SenClu: The Topic Model

SenClu is a novel inference algorithm that utilizes the fine-tuned embeddings and the BoS model.

Hard Assignments: Unlike probabilistic models (LDA) that assign soft probabilities to topics, SenClu performs hard assignments. Each sentence group is assigned to exactly one topic.
Expectation-Maximization (EM) with Annealing:
- E-Step: Assigns sentence groups to the topic with the highest probability, calculated as the product of the cosine similarity between the sentence vector and the topic vector, and the prior probability of the topic in the document.
- M-Step: Updates topic vectors by averaging the vectors of assigned sentence groups.
- Annealing: To avoid local minima, the algorithm uses a decaying smoothing parameter ( $\alpha$ ) that acts as a "temperature" control. Initially, it allows for more randomness in assignments (preventing premature convergence to poor topics) and gradually tightens the distribution.
Word-Topic Scoring: To generate interpretable topic labels, the authors propose a scoring mechanism that combines word frequency within a topic against a baseline (damped frequency) and relative importance (excess probability over random distribution). This filters out common, non-discriminatory words.

3. Key Contributions

FT-Topic Framework: An automatic, unsupervised method to fine-tune LLM encoders for topic modeling using heuristic dataset construction and quality filtering, eliminating the need for labeled data.
Bag of Sentences (BoS) Model: A novel modeling unit that bridges the gap between word-level and document-level analysis, offering a more coherent basis for topic extraction.
SenClu Algorithm: A state-of-the-art topic model that integrates BoS, hard assignments, and simulated annealing. It allows users to control topic diversity per document via a prior parameter ( $\alpha$ ).
Efficient Inference: The use of hard assignments and EM-based clustering significantly accelerates inference compared to complex deep learning topic models or variational inference methods.

4. Results and Evaluation

The authors evaluated their approach on four datasets: New York Times (NYT), 20 Newsgroups, Gutenberg, and Yelp. They compared SenClu (with and without FT-Topic) against LDA, BERTopic, and TopClus.

Topic Coherence (PMI): SenClu with FT-Topic achieved the highest Normalized Pointwise Mutual Information (PMI) scores across all datasets, significantly outperforming LDA and BERTopic. This indicates higher semantic coherence in the generated topics.
Topic Coverage (NMI): In terms of Normalized Mutual Information (NMI) against human-labeled classes, SenClu+FT-Topic consistently outperformed TopClus and other baselines.
Computational Efficiency:
- SenClu is significantly faster than TopClus (which takes hours) and comparable to BERTopic (minutes).
- FT-Topic adds training time but remains much faster than TopClus while delivering superior quality.
Qualitative Analysis: Manual inspection of topic words revealed that SenClu produces more distinct and interpretable topics. In contrast, TopClus and LDA often generated topics dominated by generic, high-frequency words (e.g., "the," "is," "please") or failed to separate distinct themes clearly.
Parameter Sensitivity: The model showed low sensitivity to hyperparameters (e.g., margin $m$ , fraction of removed data $f_{pos}/f_{tri}$ ), suggesting it is robust and easy to use with default settings.

5. Significance

This paper makes a significant contribution to the field of NLP and topic modeling by:

Bridging the Gap: It successfully integrates the power of pre-trained LLMs with the efficiency of classical topic modeling, proving that fine-tuning is essential for optimal performance even in unsupervised settings.
Paradigm Shift: It challenges the dominance of the "Bag of Words" and "Document Clustering" extremes by introducing the "Bag of Sentences," offering a more natural granularity for human topics.
Practical Utility: By providing a method that is both high-quality and computationally feasible (unlike TopClus), it offers a practical tool for researchers and practitioners. It also addresses the "black box" nature of neural topic models by providing clear sentence-to-topic assignments and interpretable word scores.
Green AI: The emphasis on reduced computation time aligns with sustainable AI practices, offering a viable alternative to energy-intensive methods.

In conclusion, the proposed FT-Topic and SenClu framework represents a state-of-the-art approach that combines unsupervised fine-tuning with a novel modeling structure to achieve superior topic coherence, coverage, and interpretability with reasonable computational costs.