Uncertainty-Aware Assessment of LLM-Enhanced Topic Models: An Experton-Based Approach to Interpretability

Technical Summary: Uncertainty-Aware Assessment of LLM-Enhanced Topic Models

Problem Statement
The study addresses three critical gaps in the evaluation of topic modeling, particularly within specialized domains like tourism where semantic ambiguity and evaluator subjectivity are prevalent:

1. Hybrid Pipeline Evaluation: Existing literature primarily compares standalone models (traditional, neural, or pure LLMs), leaving the performance of hybrid pipelines—where a neural model structures the corpus and an LLM refines the output—largely unexplored.
2. Decoding Temperature: While Large Language Models (LLMs) are increasingly integrated into topic modeling workflows, the influence of decoding temperature on both the resulting topic representations and the subsequent interpretability assessment remains under-investigated.
3. Uncertainty in Interpretability: Topic interpretability is inherently a subjective and uncertain semantic judgment. However, current evaluation methods often reduce this to deterministic scores (e.g., binary correctness or single-point coherence metrics), failing to capture the confidence intervals and uncertainty inherent in human and machine appraisal.

Methodology
The research utilized the TOURSKAD dataset, a specialized corpus of scientific tourism articles, to evaluate and compare six modeling approaches: Latent Semantic Indexing (LSI), Non-negative Matrix Factorization (NMF), Latent Dirichlet Allocation (LDA), Structural Topic Modeling (STM), Neural LDA, and BERTopic.

The methodology proceeded in two experimental rounds:

• Round 1 (Baseline Comparison): 138 models were generated across six algorithms with topic counts ($K$) ranging from 10 to 100. Models were evaluated based on computational efficiency (training time), Normalized Pointwise Mutual Information (NPMI) for intra-topic coherence, and Topic Diversity (TD) for inter-topic distinctiveness.
• Round 2 (Fine-tuning the Best Performer): BERTopic was selected as the optimal baseline. The pipeline was enhanced by:
  • Custom Embedding: Fine-tuning a ModernBERT model (designated MoST-BERT) specifically on the TOURSKAD dataset.
  • LLM Refinement: Eight different LLMs (including Mixtral-8x7b, GPT-4o, Claude-3.5-Sonnet, etc.) were tested to refine topic labels and keywords. Each LLM was evaluated across three decoding temperatures ($T=0.3, 1.0, 1.5$).
• Interpretability Assessment: A novel, uncertainty-aware framework was introduced using the Theory of Expertons and M-expertons.
  • Word-Intrusion Test: An indirect measure where judges (human coders and 15 diverse LLMs) identified semantically unrelated "intruder" words within topic lists. Precision and Krippendorff's alpha were used to measure reliability.
  • Direct Fuzzy Appraisal: Evaluators provided confidence judgments as quadruplets $(a_L, a_{C1}, a_{C2}, a_U)$ representing support intervals and maximum presumption zones. These were aggregated into M-expertons to preserve uncertainty.
  • Counter-Appraisement: Independent panels of LLMs (Panel A, B, and C) were compared against each other and against human coders using expectation distance and Hamming distance metrics to validate convergence.

Key Results

• Model Performance: The LLM-enhanced BERTopic pipeline (specifically using MoST-BERT embeddings and Mixtral-8x7b for refinement) outperformed all traditional and neural baselines. It achieved the strongest combined performance in coherence and diversity, particularly at $K=30$.
• Temperature Sensitivity: Decoding temperature was found to be a substantive modeling parameter rather than a mere technical setting. Mixtral-8x7b at $T=1.0$ yielded the optimal balance of coherence (NPMI=0.45) and diversity (0.83). While some models (e.g., Gemini-1.5-Pro) performed well at higher temperatures, precision in word-intrusion tasks generally decreased as temperature increased, though this varied by model architecture.
• Interpretability Metrics:
  • Word-Intrusion: Human coders achieved the highest precision ($MP=0.892$). LLM panels followed closely ($MP \approx 0.820-0.831$), with Gemma 4 31B and GPT-5.5 performing best among LLMs.
  • M-Expertons: The M-experton framework successfully aggregated judgments while preserving uncertainty. Human coders exhibited narrower support widths (mean 0.193) compared to LLM panels (mean 0.259), indicating that LLMs expressed greater imprecision in their confidence intervals, particularly in the outer support bounds rather than the core presumption.
  • Convergence: M-experton appraisals showed higher alignment with human judgments than word-intrusion scores. However, divergence was observed in specific topics (e.g., "Emotional Intelligence," "Accessible Tourism"), where LLM expectations deviated more significantly from human benchmarks.

Key Contributions

1. Benchmarking Hybrid Pipelines: The study provides empirical evidence that a hybrid pipeline (BERTopic for clustering + LLM for refinement) outperforms standalone traditional and neural models in specialized tourism corpora, offering a methodologically traceable alternative to pure LLM extraction.
2. Redefining Temperature: The paper establishes decoding temperature as a critical hyperparameter that directly influences topic representation quality and interpretability assessment, urging researchers to treat it as a modeling variable.
3. Uncertainty-Aware Evaluation Framework: The study introduces the first application of the Theory of Expertons (via M-expertons) to topic modeling. This framework allows for the aggregation of human and LLM judgments without collapsing uncertainty into a single deterministic score, enabling a more nuanced assessment of interpretability that accounts for evaluator subjectivity and confidence.

Significance
The paper claims that its proposed framework is particularly relevant for specialized corpora where topic interpretation involves semantic ambiguity and uncertain boundaries between themes. By operationalizing interpretability as a subjective and uncertain appraisal, the study offers a more robust evaluation method for specialized domains like tourism, where standard deterministic metrics may fail to capture the complexity of human understanding. The findings suggest that while LLMs can approximate human interpretability, they do so with different uncertainty profiles, necessitating evaluation methods that explicitly account for these differences.