💬 NLP

Improving Topic Modeling by Distilling Soft Labels from Language Models

This paper proposes a novel topic modeling framework called Distilling Soft Labels (DSL) that leverages language models to generate contextually enriched soft labels for training, thereby significantly improving topic coherence, assignment accuracy, and document retrieval performance compared to traditional methods.

Original authors: Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive library of books, but instead of reading the whole story, you only look at the list of words that appear on the cover. Traditional computer programs do exactly this: they count how often words like "doctor," "pain," or "treatment" appear together. If a book mentions "pain" and "doctor" often, the computer groups them under a "Health" topic.

The problem? This method is like trying to understand a movie by only reading the cast list. It misses the story, the context, and the deeper meaning. If a short text talks about a "broken bone" but never uses the word "doctor," the old computer might miss the medical theme entirely.

The New Approach: The "Smart Assistant" Teacher

The authors of this paper, Raymond Li and his team, propose a clever new way to teach computers to understand topics. Instead of just counting words, they use a "Smart Assistant" (a Small Language Model or SLM) to act as a teacher.

Here is how their method, called DSL (Distilling Soft Labels), works, using a simple analogy:

1. The "Guess the Theme" Game

Imagine you hand a short, confusing note to a very smart, well-read assistant and say, "What is the main theme of this note?"

Old Way: The computer looks at the note and says, "It has the word 'hockey' in it, so the topic is Hockey."
New Way (DSL): The smart assistant reads the note and thinks, "This is about a trade between sports teams. Even though the word 'hockey' isn't written here, the context screams 'sports' and 'team management'."

2. The "Soft" Answer (The Secret Sauce)

The smart assistant doesn't just shout out one word like "Hockey." Instead, it gives a soft, fuzzy list of probabilities.

It says: "There's a 40% chance this is about hockey, a 30% chance it's about sports, a 20% chance it's about trades, and a 10% chance it's about news."
This is called a Soft Label. It captures the vibe and the hidden themes of the text, even if the specific words aren't there.

3. The Student Learns from the Teacher

The researchers then take this "fuzzy list" from the smart assistant and use it to train a simpler, faster computer program (the Topic Model).

The Training: They tell the simple program: "Don't just guess the words you see. Try to guess the whole list of themes that the smart assistant gave you."
The Result: The simple program learns to look deeper. It realizes that even if the word "hockey" is missing, the presence of "trade," "players," and "draft" strongly suggests the topic is hockey.

Why This Matters (The Results)

The paper tested this on three different types of text collections:

20Newsgroups: Old internet forum posts.
TweetTopic: Short tweets.
StackOverflow: Short coding questions.

The Findings:

Better Understanding: The new method created topics that made much more sense to humans. For example, in a sports discussion, it correctly identified "hockey" as a theme even when the word "hockey" didn't appear in the text, because the context was so clear.
Finding Similar Documents: If you asked the system to find documents similar to a specific one, it was much better at finding the right matches than previous methods. It was like having a librarian who understands the plot of a book, not just the index words.
Short Texts: It worked especially well for short texts (like tweets), where there are very few words to count, making the old methods fail.

The "Teacher" Doesn't Need to Be a Giant

One of the coolest parts of this paper is that they didn't need a massive, super-expensive AI to be the teacher. They used Small Language Models (SLMs)—which are like compact, efficient versions of the giant AI models. These small models were fast, cheap to run, and still smart enough to teach the topic model how to understand context.

In a Nutshell

The paper introduces a way to teach computers to understand the meaning behind text, not just the words on the page. By using a smart AI teacher to give "fuzzy hints" about what a text is really about, they trained a simpler system to organize documents much more accurately. It's like upgrading from a computer that counts words to one that actually reads for comprehension.

Technical Summary: Improving Topic Modeling by Distilling Soft Labels from Language Models

Problem Statement
Traditional neural topic models (NTMs) are typically optimized by reconstructing a document's Bag-of-Words (BoW) representation. While effective for capturing statistical word co-occurrences, this approach overlooks rich contextual information and struggles significantly with data sparsity, particularly in short-text corpora. Conversely, while Large Language Models (LLMs) possess deep semantic understanding, they face limitations in modeling large corpora due to context-length constraints and computational costs. Existing attempts to leverage LLMs for topic modeling often rely on complex, costly pipelines or prompt-based generation that fails to produce the probabilistic topic distributions required for downstream applications like uncertainty-aware retrieval.

Methodology: Distilling Soft Labels (DSL)
The authors propose a novel training framework, Distilling Soft Labels (DSL), which bridges the gap between the semantic capabilities of Small Language Models (SLMs) and the structural efficiency of probabilistic topic models. The framework operates through three primary components:

Semantic Reconstruction Targets: Instead of using sparse BoW targets, DSL constructs dense, contextually enriched reconstruction signals. Given an input document $x$ and a thematic instruction prompt $\pi$ (e.g., "Generate a single-word label capturing the document theme"), the SLM predicts the next-token distribution. The logits for the next token are projected onto a predefined vocabulary $V$ and converted into a temperature-scaled softmax distribution, denoted as $y_{DSL}(x, \pi)$ . Unlike BoW, which only assigns mass to words explicitly present in the document, $y_{DSL}$ assigns probability mass to thematically relevant words even if they do not appear in the text.
Contextualized Inputs: The topic model utilizes the final hidden state representation of the SLM (at the last position of the prompt-conditioned input) as the document embedding. This ensures the input is naturally aligned with the semantic distribution generated by the model.
Training Objective: The neural topic model is trained to reconstruct the dense $y_{DSL}$ target rather than the sparse BoW target. The objective function minimizes the Kullback-Leibler (KL) divergence between the predicted vocabulary distribution $\hat{y}_\psi(x)$ and the SLM-induced target $y_{DSL}(x, \pi)$ , alongside model-specific regularization terms (e.g., variational priors or embedding clustering):
$L_{DSL}(x) = \lambda D_{KL}(y_{DSL}(x, \pi) \parallel \hat{y}_\psi(x)) + R_M(x; \psi)$

Bayesian Interpretation
The authors interpret DSL through the lens of implicit Bayesian inference. They posit that the SLM acts as an implicit Bayesian predictor over latent concepts. The projected next-token distribution $y_{DSL}$ represents an implicit posterior predictive distribution over theme-relevant vocabulary. The DSL framework effectively distills this unstructured, implicit signal into an explicit, structured probabilistic topic model, yielding interpretable document-topic and topic-word distributions.

Key Contributions

Novel Training Paradigm: The introduction of a training objective that uses semantically grounded soft label distributions derived from SLMs as the reconstruction target for neural topic models.
Performance Improvements: Extensive experiments demonstrate that DSL achieves substantial improvements over existing baselines (including LDA, ProdLDA, BERTopic, ECRTM, and FASTopic) in topic coherence, diversity, and assignment accuracy across three datasets (20NewsGroup, StackOverflow, TweetTopic).
Retrieval-Based Evaluation: The authors introduce a retrieval-based metric to evaluate the utility of topic distributions for finding semantically similar documents. Results indicate that DSL significantly outperforms baselines in retrieval precision, particularly for short-text datasets where data sparsity is a major challenge.

Experimental Results

Topic Quality: Across various backbone architectures (ProdLDA, ECRTM, FASTopic) and SLM teachers (Llama, Phi, Qwen, ERNIE), DSL consistently outperforms baselines. For instance, on the StackOverflow dataset, the Purity score for ProdLDA improved from 0.265 to 0.803 when enhanced with DSL.
Robustness: The method remains effective regardless of the specific SLM teacher used, provided the SLM follows instructions well.
Retrieval Performance: In retrieval tasks, DSL-based models significantly outperform baselines. On StackOverflow, the best DSL variant nearly tripled the baseline precision at 5 (P@5), rising from 0.306 to 0.867.
Ablation Studies: The authors confirm that the primary driver of performance is the semantically grounded soft target; replacing the KL loss with standard NLL or reverting to BoW targets results in significant performance drops. The use of SLM hidden states as input is shown to be beneficial specifically for reconstructing soft targets, rather than for standard BoW reconstruction.

Significance and Claims
The paper claims that DSL offers a simple, efficient, and effective paradigm for enhancing neural topic modeling without the computational overhead of full LLM inference pipelines or the limitations of sparse BoW representations. By distilling the "thematic posterior predictive signal" of an SLM into a probabilistic topic model, the approach preserves the interpretability and uncertainty-awareness of traditional topic models while significantly improving their ability to capture underlying thematic structures. The authors position this work as a step toward better leveraging SLMs for retrieval-oriented applications, noting that the resulting low-dimensional topic representations offer computational and storage advantages over high-dimensional dense embeddings.

The authors remain modest regarding future directions, suggesting that future work could explore different prompting strategies (e.g., few-shot learning, chain-of-thought) and multi-modal applications, but they do not claim the method solves all challenges in topic modeling or that it is a definitive solution for all downstream tasks. They also acknowledge potential societal risks related to bias inheritance from the distilled language models.