💬 NLP

Repeatability is not recovery: Quantifying algorithmic stability and topic recovery in Latent Dirichlet Allocation

This paper demonstrates that the repeatability of Latent Dirichlet Allocation (LDA) outputs across repeated runs does not guarantee the accurate recovery of underlying topics, arguing that internal stability and ground-truth recovery are distinct properties that must be evaluated separately to avoid misleading conclusions.

Original authors: Saranzaya Magsarjav, Jonathan Tuke, Lewis Mitchell, Melissa Humphries

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saranzaya Magsarjav, Jonathan Tuke, Lewis Mitchell, Melissa Humphries

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, unlabelled oceans of human writing, from ancient manuscripts to modern social media posts, there are hidden patterns waiting to be found. These patterns are groups of words that frequently appear together, forming the invisible themes or "topics" that give a collection of text its meaning. For decades, scientists have used mathematical tools to sift through these mountains of words, hoping to pull out these hidden themes automatically. The goal is to let a computer read a library of documents and tell us, "Here are the main subjects being discussed." This process is called topic modelling, and it has become a standard way to understand everything from historical archives to medical records. However, because the computers use a bit of randomness to find these patterns, running the same analysis twice on the same text can sometimes yield slightly different results. For a long time, researchers assumed that if a computer kept finding the same themes over and over again, it must have found the "true" themes hidden in the data.

A team of researchers at Adelaide University has challenged this assumption, showing that just because a computer is consistent, it does not mean it is correct. They set out to test whether the ability of a topic model to repeat its own findings is the same thing as its ability to recover the actual truth. To do this, they created a controlled experiment where they knew the answer before they even started. They generated fifty different sets of simulated text, each built from a known, hidden structure of ten distinct topics. They then ran their topic-finding algorithm on these texts fifty times each, using different random starting points for every run. By comparing the results of these repeated runs against the known truth they had built into the simulation, they could see exactly how well the computer was doing.

The researchers discovered a surprising gap between consistency and accuracy. They found that the algorithm was very good at being consistent; when they ran it multiple times, it almost always produced the same set of topics. However, those repeated topics were not always the same as the true topics they had planted in the data. In many cases, the computer was confidently and repeatedly finding the wrong answer. It was as if the algorithm had learned to find a specific pattern that looked like a topic, but it was not the actual topic the data was built on. This distinction is crucial because in the real world, where we do not know the hidden truth, we often rely on repeatability as a sign of quality. This study shows that repeatability is a measure of how stable the computer's process is, not a guarantee that it has found the right answer.

To get a clearer picture, the team looked at the data in three different ways. First, they compared the full list of words and their probabilities for every topic. Second, they looked only at the top ten most important words for each topic, which is how humans usually read and interpret these results. Third, they checked the order in which those top words appeared. They found that the algorithm was much better at finding the correct top words and their order than it was at getting the full mathematical distribution right. For the clearest, most separated topics, the computer could accurately identify the key words and their ranking. But for topics that were more mixed or overlapping, the computer's repeated results were consistent with each other, yet they still drifted away from the true structure.

The researchers also tested their method on a real-world dataset of twenty thousand messages from twenty different online discussion groups, ranging from religion to sports. Since they did not know the exact mathematical structure of this real data, they could not measure "recovery" in the same way, but they could still measure consistency. They found that the measures of consistency were lowest when the number of topics matched the twenty known categories of the discussion groups. This suggests that the algorithm was most stable when it was trying to find a number of topics that aligned with the real-world structure of the data. However, other measures of quality, which looked at how coherent the words within a topic seemed, pointed toward different numbers of topics. This reinforces the idea that different ways of measuring success can tell different stories.

The ultimate takeaway is that in the world of automated text analysis, a stable result is not necessarily a correct one. A topic model can be highly repeatable, producing the same themes every time it runs, while still missing the true underlying structure of the data. The researchers conclude that scientists and analysts should not treat a repeatable output as proof that the topics are correct. Instead, they must look at repeatability, accuracy, and the coherence of the words together. If a model is consistent but the topics do not make sense, or if the words are coherent but the model is unstable, the results are incomplete. By understanding that repeatability is not the same as recovery, we can use these powerful tools more carefully, knowing that finding the same pattern twice does not mean we have found the truth.

Technical Summary: Repeatability is not recovery: Quantifying algorithmic stability and topic recovery in Latent Dirichlet Allocation

Problem Statement
Topic modeling, particularly using stochastic algorithms like Latent Dirichlet Allocation (LDA), is widely used to uncover hidden structures in unlabelled text. A common practice in evaluating these models is to rerun them multiple times; if the outputs are consistent (repeatable), this is often interpreted as evidence that the model has successfully recovered the true underlying topics. However, this paper argues that repeatability does not equate to recovery. A model may consistently produce similar, yet incorrect, topic structures due to the stochastic nature of inference, random initialization, or corpus sampling variations. The core problem is the lack of a framework that simultaneously quantifies the consistency of repeated runs (repeatability) and the accuracy of those runs relative to a known ground truth (recovery), especially given the scarcity of high-quality labelled data for real-world corpora.

Methodology
To address this, the authors propose an Instability framework that combines recovery accuracy with consistency across repeated fits. The methodology involves three primary components:

Simulated Ground Truth: The authors generated 50 distinct simulated corpora using the LDA generative process with a known number of topics ( $K=10$ ). These corpora utilized five different topic-word distributions to vary the degree of topic separation:
- Dirichlet distributions (parameters 0.0001 and 0.001).
- Normal distributions (standard deviations 1 and 10).
- Uniform Separable distribution (completely non-overlapping topics).
  Each corpus was fitted 50 times using fixed random seeds to capture variability from both corpus sampling and model fitting.
Similarity Measures: Three distinct metrics were employed to compare topic outputs against the ground truth and against each other:
- Jensen–Shannon Similarity (JSS): Measures the distance between complete topic-word probability distributions.
- Jaccard Similarity (JIS): Measures the overlap of the top 10 terms (treated as unordered sets).
- Rank-Biased Overlap (RBO): Measures the similarity of the top 10 terms while accounting for their ordering.
The Instability Metric: The authors define Instability as the Euclidean distance between a point $(\bar{m}, \text{var}(m))$ and the ideal point $(1, 0)$ , where $\bar{m}$ is the mean similarity score and $\text{var}(m)$ is the variance across repeated runs.
- When comparing to ground truth, low Instability indicates high recovery accuracy and high consistency.
- When comparing repeated runs to each other, low Instability indicates high internal repeatability.
- The metric is calculated as: $\text{Instability}_t = \frac{1}{t} \sum_{i=1}^{t} \sqrt{(\bar{m}_i - 1)^2 + \text{var}(m_i)^2}$ .

The framework was also applied to the 20 Newsgroups dataset (where the ground truth is unknown but 20 labelled categories exist) to test the measures in a real-world scenario, comparing them against NPMI (Normalized Pointwise Mutual Information) topic coherence.

Key Contributions

Conceptual Distinction: The paper establishes that repeatability and recovery are distinct properties. A model can be highly repeatable (low variance in outputs) while failing to recover the true generating distribution (low mean similarity to ground truth).
Instability Framework: The introduction of a unified metric that utilizes mean and variance of similarity scores to simultaneously assess accuracy and consistency.
Controlled Data Generation: The creation of a simulation pipeline with known ground truths and varying levels of topic overlap, allowing for a direct evaluation of model performance that is impossible with standard real-world datasets.

Results

Repeatability vs. Recovery: Across most distributions, LDA was found to be more repeatable than accurate. The Instability values for repeated runs (consistency) were generally lower than those comparing outputs to the ground truth (accuracy). This indicates that LDA often converges to a consistent, but potentially incorrect, solution.
Distribution Sensitivity:
- For the Uniform Separable distribution, LDA successfully recovered both the top terms and their ordering (low Instability for JIS and RBO against ground truth), and NPMI coherence aligned with the Instability measures at $K=10$ .
- For overlapping distributions (Dirichlet and Normal), the signal for the true topic count ( $K=10$ ) was weaker. NPMI coherence generally favored smaller topic counts, whereas the Instability measures showed a minimum or plateau near $K=10$ .
Metric Differences:
- Jensen–Shannon Similarity (full distributions) showed the largest gap between repeatability and recovery, indicating that while the full probability distributions are not accurately recovered, the models are consistent with each other.
- Jaccard and RBO (top terms) showed better recovery for the Uniform Separable distribution, suggesting that while the model identifies the correct representative words, it may not estimate their full probability distributions accurately.
20 Newsgroups Application: On the real-world dataset, Jaccard and RBO Instability measures showed a minimum near the 20 labelled categories, suggesting the top terms are most repeatable at this count. However, Jensen–Shannon Instability and NPMI showed different patterns, reinforcing that different metrics capture different aspects of model performance.

Significance and Claims
The authors claim that their results demonstrate that repeatability should not be treated as evidence that fitted topics are correct. Instead, repeatability is merely evidence that the fitting procedure yields a similar solution. The paper argues that:

Topic model validation requires multiple forms of evidence: repeated fitting (to assess uncertainty), coherence measures, and, where possible, external ground truth.
The choice of similarity measure matters: Jaccard and RBO are relevant for qualitative interpretation (labeling topics), while Jensen–Shannon is necessary for quantitative analysis involving full distributions.
The "true" number of topics is not solely defined by the minimum Instability value; rather, Instability identifies a "comparatively repeatable" solution, and the interpretation of that solution depends on the specific structure of the corpus and the analytical goals.

The paper concludes that while the proposed framework provides a controlled method to distinguish between repeatability and recovery, real-world applications require caution, as high coherence or repeatability does not guarantee the recovery of the underlying generative structure.

Technical Summary: Repeatability is not recovery: Quantifying algorithmic stability and topic recovery in Latent Dirichlet Allocation

More like this