💬 NLP

Determination of the Number of Topics Intrinsically: Is It Possible?

This paper evaluates various intrinsic methods for estimating the number of topics in topic models across multiple corpora and concludes that these methods are unreliable because the optimal number of topics is dependent on the specific model and method used rather than being an absolute property of the data.

Original authors: Victor Bulatov, Vasiliy Alekseev, Konstantin Vorontsov

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Victor Bulatov, Vasiliy Alekseev, Konstantin Vorontsov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a library containing millions of books, but the shelves are empty and the titles are missing. To make sense of this collection, a librarian might try to group the books by their content, sorting them into piles based on shared themes. In the world of computer science, this is the job of a topic model. These are statistical tools that scan vast amounts of text to find hidden patterns, grouping words that frequently appear together into what we call "topics." A topic might be about sports, another about cooking, and another about space exploration. The computer does not know these labels beforehand; it discovers them by analyzing the text itself.

However, there is a fundamental problem with this process: the computer does not know how many piles to make. Should it create ten groups, fifty, or a hundred? This number is a crucial setting, or "hyperparameter," that the user must choose before the computer begins its work. If the number is too low, the groups become messy and confusing, mixing unrelated subjects. If the number is too high, the groups become tiny and repetitive, splitting a single idea into many nearly identical fragments. For years, researchers have tried to build automatic tools that could look at a collection of text and tell the user the perfect number of topics to choose, hoping to find a single, objective truth hidden within the data.

A team of researchers from the Moscow Institute of Physics and Technology and Lomonosov Moscow State University decided to test whether such a perfect number actually exists. They gathered a wide variety of existing methods—dozens of different mathematical formulas and procedures—that claim to determine the best number of topics. They applied these methods to several large, real-world collections of text, ranging from news articles and scientific papers to posts from a programming forum and a curated set of Russian Wikipedia articles. They also tested these methods on different types of computer models, as the way a model is built can change how it sees the data.

The researchers ran thousands of experiments, letting the computer models learn from the text while measuring the results with each of the different methods. They looked for clear signals, such as a sharp peak or a deep valley in the data, that would indicate the ideal number of topics. They hoped to find that all the different methods would agree with each other, pointing to the same number for a given collection of text. Instead, they found a chaotic picture. The different methods rarely agreed. One method might suggest that a collection of news articles is best understood with ten topics, while another method, looking at the exact same text, would suggest thirty. Even more surprisingly, the "best" number often changed depending on which specific computer model was used to analyze the text.

The study revealed that the most popular and sophisticated tools for finding the right number of topics are far from reliable. Some methods, which try to measure how distinct the topics are from one another, consistently suggested numbers that were far too high, often pushing the count to the very limit of what the researchers tested. Other methods, which look at how well the model predicts unseen text, produced results that were so flat and unchanging that they offered no guidance at all. The researchers found that the number of topics is not a fixed, natural property of the text itself, like the number of pages in a book. Rather, it is a quantity that depends heavily on the specific tool used to measure it and the specific model used to analyze it.

In their conclusion, the authors suggest that the community has been chasing a myth. The idea that there is a single, "natural" number of topics waiting to be discovered in any dataset appears to be incorrect. The number of topics is more like a dial that a user turns to adjust the level of detail they want to see, similar to zooming in or out on a map. A map of a city can show individual streets, or it can show only major highways; neither is the "true" map, they are just different views for different purposes. The researchers argue that instead of trying to find a perfect number automatically, practitioners should focus on what they actually need the model to do. They should ask how detailed they want the groups to be, or whether they have specific questions they want the model to answer. The search for an intrinsic, automatic solution to the number of topics has, according to this study, led to a dead end, suggesting that the answer lies not in a better formula, but in a better understanding of the human goal behind the analysis.

Technical Summary: Determination of the Number of Topics Intrinsically: Is It Possible?

Problem Statement
The number of topics ( $T$ ) is the primary hyperparameter in topic modeling, yet there is no consensus on how to determine its optimal value. While numerous intrinsic methods exist to estimate $T$ without external labels or human assessment, the literature lacks a comprehensive comparison of these practices across different models and corpora. The authors investigate whether "optimal $T$ " is an absolute, intrinsic property of a corpus or if it is merely a model-dependent quantity. They specifically exclude methods that automatically learn $T$ (e.g., Hierarchical Dirichlet Processes) and focus on fixed- $T$ models where $T$ must be selected via quality metrics.

Methodology
The study evaluates a wide range of intrinsic quality metrics across multiple topic models and publicly available corpora.

Topic Models: The experiments utilized PLSA, LDA (with symmetric, asymmetric, and heuristic priors), decorrelated models (TWC-LDA), sparse models (background/specific topic separation), and thetaless models (TARTM). These were implemented using the BigARTM/TopicNet stack.
Corpora: Seven datasets were analyzed, including WikiRef220, 20NewsGroups, Brown Corpus, Reuters, StackOverflow, PostNauka, and a new "ruwiki-good" dataset derived from Russian Wikipedia. These datasets vary in size (220 to ~900k documents) and have varying "ground truth" or expected topic ranges.
Metrics Evaluated: The study assessed metrics from several categories:
- Perplexity: Hold-out perplexity and Rate of Perplexity Change (RPC).
- Stability: Subsample-based stability (Jaccard similarity of top tokens) and cophenetic correlation.
- Diversity: Average and closest-topic distances using Cosine, Jensen-Shannon, Euclidean, and Hellinger metrics (e.g., D-avg-COS, D-cls-JS).
- Information-Theoretic: AIC, BIC, and MDL (both standard and sparse variants).
- Entropy: Renyi entropy based on energy states.
- Other: Lift-score, Silhouette Coefficient, Calinski-Harabasz Index, and Average Coherence.
Experimental Design: Models were trained with $T$ ranging from a minimum to a maximum value across three random initializations. Metrics were calculated for each run. To determine if a metric yields a reliable "optimal" $T$ , the authors algorithmically searched for pronounced global minima/maxima and checked for robustness across random seeds.

Key Results
The experimental results, summarized in Table 3, indicate that intrinsic methods are generally unreliable for determining a single, absolute number of topics.

Model Dependency: The "optimal" $T$ varies significantly depending on the specific topic model and its hyperparameter configuration (e.g., priors in LDA). A value optimal for one model is often suboptimal for another.
Metric Disagreement: Different metrics frequently suggest vastly different values for $T$ . Diversity-based methods tend to overestimate $T$ (often suggesting values outside the tested range), while information-theoretic methods (AIC, BIC, MDL) and entropy-based methods (Renyi) showed relatively better performance, though still inconsistent.
Randomization Sensitivity: Many metrics produce different optimal values based on random initialization seeds. While some methods (like Information-Theoretic ones) show stable optima across seeds, others (like Diversity and Coherence) exhibit high variance or noisy curves with multiple local optima.
Specific Metric Performance:
- Perplexity: Often monotonic without clear minima, contradicting earlier literature.
- Stability: Often too noisy for small $T$ and unsuitable for sparse models.
- Coherence: Generally a declining function of $T$ , leading to the selection of very small $T$ values.
- Diversity: Often fails to show a clear maximum, instead flattening out.
- Simplest Methods: Surprisingly, the simplest approaches (AIC, BIC, MDL, Renyi entropy) provided the most consistent, albeit not perfect, results compared to complex, structure-based metrics.

Significance and Conclusions
The paper concludes that the notion of an "optimal number of topics" as an intrinsic, absolute property of a corpus is likely a myth. Instead, the number of topics is shown to be a model-dependent quantity, influenced by the specific algorithm, its hyperparameters, and the chosen metric for evaluation.

The authors argue that the community's search for a "natural" number of topics inherent in data is a subject of ongoing debate. They suggest that $T$ should be treated as a standard machine learning hyperparameter to be tuned based on the specific goals of the application rather than an intrinsic truth to be discovered.

Proposed Directions:
Rather than relying on intrinsic metrics to find a "true" $T$ , the authors suggest:

Treating $T$ as a tunable hyperparameter.
Focusing on secondary tasks (e.g., classification) to select $T$ .
Developing models that are robust to changes in $T$ or capable of learning hierarchical topic structures.
Shifting the practitioner's focus from "finding the number of topics" to defining desired granularity (e.g., "how many documents should a topic contain?") and ensuring topic interpretability.

Ultimately, the study posits that the goal of topic modeling should not be the determination of an optimal $T$ , but the development of training methods that yield interpretable topics for a given $T$ in the absence of external criteria.

More like this