Topeax -- An Improved Clustering Topic Model with Density Peak Detection and Lexical-Semantic Term Importance

Technical Summary of "Topeax: An Improved Clustering Topic Model with Density Peak Detection and Lexical-Semantic Term Importance"

Problem Statement
While text clustering has become the dominant paradigm for topic modeling in both academia and industry, existing state-of-the-art models, specifically Top2Vec and BERTopic, exhibit significant limitations. The author identifies two primary issues:

1. Unreliable Cluster Discovery: These models are highly sensitive to sample size and hyperparameters (such as perplexity). Their default configurations often result in suboptimal behavior, failing to discover natural clusters in corpora without manual intervention (e.g., hierarchical reduction to a user-specified number of topics).
2. Suboptimal Term Importance Estimation: Current methods for generating topic keywords are flawed. BERTopic ignores the semantic distance of keywords to topic vectors, while Top2Vec ignores word counts in the corpus. Consequently, Top2Vec often includes stop words and "junk" words, reducing topic coherence, while BERTopic may lack variety and trust in its keyword selection.

Methodology: The Topeax Model
The paper introduces Topeax, a novel clustering topic model implemented in the Turftopic Python package. Topeax addresses the identified issues through a three-stage pipeline:

1. Dimensionality Reduction:
   Unlike Top2Vec and BERTopic, which utilize UMAP, Topeax employs t-Distributed Stochastic Neighbour Embedding (t-SNE). The model uses cosine distance as the metric and fixes the output dimensionality to 2 to facilitate visualization and density estimation.

2. The Peax Clustering Algorithm:
   Topeax introduces a new clustering technique called Peax, which determines the number of clusters automatically based on density peaks rather than relying on user-defined parameters or hierarchical merging. The algorithm proceeds as follows:

   • Density Estimation: A Gaussian Kernel Density Estimate (KDE) is computed over the reduced embeddings, with bandwidth determined by the Scott method.
   • Grid Evaluation: The KDE is evaluated on a 100x100 grid covering the embedding range.
   • Peak Detection: Density peaks are identified using a local-maximum filter with a neighborhood connectivity of 25 (a 5-unit radius).
   • Cluster Assignment: These peaks serve as cluster centers. A Gaussian Mixture Model (GMM) is fitted with means fixed to these peaks. Documents are assigned to the component with the highest responsibility ($r_{kd}$) using the Expectation-Maximization algorithm.

3. Hybrid Term Importance Estimation:
   Topeax combines lexical and semantic indices to generate high-quality keywords, avoiding the pitfalls of pure centroid proximity or pure TF-IDF weighting.

   • Semantic Importance: Topic vectors ($t_k$) are calculated as the responsibility-weighted average of document embeddings within a cluster. Semantic importance ($s_{kj}$) is the cosine similarity between the term embedding and the topic vector.
   • Lexical Importance: Instead of c-TF-IDF, Topeax uses Normalized Pointwise Mutual Information (NPMI). This measures the mutual information between a term's presence and the cluster label. To prevent overestimating rare words, the model employs a mean-a-posteriori estimate under a symmetric Dirichlet prior with a smoothing parameter ($\alpha=2$).
   • Combination: The final importance score ($\beta_{kj}$) is the geometric mean of the min-max normalized lexical (NPMI) and semantic scores.

Experimental Setup
The evaluation utilized a "free-clustering" scenario where models were not given the number of clusters in advance.

• Datasets: The study used gold-labeled datasets from the Massive Text Embedding Benchmark (MTEB), an annotated Twitter topic-classification dataset, and a BBC News dataset.
• Baselines: Topeax was compared against Top2Vec and BERTopic using various sentence encoders (MiniLM, MPNet, and EmbeddingGemma).
• Metrics:
  • Cluster Recovery: Measured via the Fowlkes-Mallows Index (FMI) to assess alignment with gold labels.
  • Topic Quality: Evaluated using topic diversity ($d$), external word embedding coherence ($C_{ex}$), and internal coherence ($C_{in}$), aggregated into an interpretability score ($I$).
• Sensitivity Tests: The models were tested for sensitivity to perplexity values and subsampling invariance (performance on partial vs. full corpora).

Results
The experimental results demonstrate that Topeax substantially outperforms both Top2Vec and BERTopic in two key areas:

1. Cluster Recovery: Topeax achieved a significantly higher Fowlkes-Mallows Index, indicating a better match with ground-truth cluster labels. A regression analysis confirmed that the model type was a significant predictor of performance ($p=0.0169$), with both BERTopic and Top2Vec showing significantly negative slopes compared to Topeax.
2. Topic Quality: Topeax generated higher-quality topic keywords, resulting in better coherence and interpretability scores.
3. Stability: Unlike the baselines, Topeax exhibited less erratic behavior when subjected to changes in sample size (subsampling) and hyperparameters (perplexity).

Significance and Claims
The paper claims that Topeax offers a more robust and reliable approach to topic modeling by solving the "black box" nature of cluster number determination in existing models. By leveraging density peak detection (Peax) and a hybrid lexical-semantic importance metric, Topeax reduces the need for manual hyperparameter tuning and hierarchical topic reduction. The author asserts that this approach yields more coherent topics and more accurate recovery of natural data structures, making it a superior alternative for both academic and industrial applications where automatic, high-quality topic discovery is required.