PRISM: LLM-Guided Semantic Clustering for High-Precision Topics

1. Problem Statement

The paper addresses the challenge of identifying distinct, nuanced narratives within large text corpora (e.g., social media posts) using topic modeling.

• Limitations of Classical Models: Traditional models like LDA rely on bag-of-words assumptions and count statistics, failing to capture subtle semantic differences in short or domain-specific texts.
• Limitations of Embedding Models: Modern embedding-based models (e.g., Top2Vec, BERTopic) utilize semantic vector spaces but often lack local precision. They struggle to distinguish between closely related sub-topics within a narrow domain.
• Limitations of LLM-Driven Approaches: Recent methods use Large Language Models (LLMs) to guide clustering for higher quality. However, relying on LLMs during the inference/clustering phase creates a dependency that results in high latency, high costs, and poor scalability.

Goal: To create a framework that achieves the high semantic resolution of LLMs for topic discovery while maintaining the low cost, speed, and interpretability of lightweight, locally deployable models.

2. Methodology: PRISM Framework

PRISM (Precision-Informed Semantic Modeling) is a student-teacher framework that distills the semantic capabilities of a large LLM into a lightweight sentence encoder.

A. Architecture Overview

1. Teacher Model ($L$): A large, expensive embedding or generative model (e.g., OpenAI's text-embedding-3-large or gpt-5) used only during the training phase to generate labels.
2. Student Model ($S$): A lightweight, pre-trained sentence encoder (initially all-mpnet-base-v2 from SentenceTransformers) that is fine-tuned to mimic the teacher's semantic understanding.
3. Clustering Algorithm ($A$): A thresholded community detection algorithm (from SentenceTransformers) that groups items based on similarity thresholds, allowing for unassigned items (singletons) to preserve nuance rather than forcing all data into clusters.

B. Training Process (Knowledge Distillation)

The core innovation lies in how the student model is fine-tuned using data labeled by the teacher. The authors propose three sampling strategies to construct the training dataset ($D$):

1. Binary Comparison (Range-Bound - $D_{RB}$):
   • The teacher LLM is queried with a zero-shot prompt to determine if two text items are semantically identical ("Yes/No").
   • Crucial Step: Pairs are sampled such that their initial similarity (via the un-tuned student) falls within a "hard" range (e.g., 0.65–0.95). This forces the student to learn distinctions in the most ambiguous regions of the embedding space.
2. Binary Comparison (Full-Range - $D_{FR}$): Random pairs are sampled without restriction.
3. Embedding Similarity ($D_{Emb}$): The teacher embedding model generates embeddings for a set of items, and pairwise similarity scores are calculated directly to form regression-style training triples.

Loss Function: The student model is fine-tuned using CoSENT loss, which the authors found superior to standard contrastive losses for these tasks.

C. Inference

Once fine-tuned, the student model $S$ encodes the entire corpus. Clustering is performed using a thresholded algorithm on these embeddings. No LLM calls are required at inference time, ensuring zero latency and cost during deployment.

3. Key Contributions

1. Student-Teacher Pipeline for Topic Discovery: A novel method to distill sparse, high-quality LLM supervision into a lightweight encoder, enabling high-precision topic modeling without runtime LLM dependency.
2. Sampling Strategy Analysis: An empirical analysis showing that Range-Bound (RB) sampling (focusing on hard-to-distinguish pairs) significantly improves the local geometry of the embedding space, leading to better cluster separability.
3. Thresholded Clustering for Precision: The use of thresholded clustering that allows for "unassigned" items prevents the collapse of nuanced topics into broad, inaccurate clusters, a common failure mode in standard partitioning algorithms.
4. Web-Scale Applicability: Demonstrates a framework suitable for tracking nuanced claims and subtopics online with an interpretable, locally deployable system.

4. Experimental Results

The authors evaluated PRISM on three diverse corpora:

• HumAID: 77K disaster-related tweets (Human-labeled events).
• LIAR: 12.8K political statements labeled for truthfulness.
• IMDB: 50K movie reviews (sentiment analysis).

Baselines:

• Top2Vec and BERTopic (using all-mpnet-base-v2).
• Direct clustering on the large teacher embedding ($A \circ L_{Emb}$).

Key Findings:

• Superior Discrimination: Fine-tuning the student model significantly increased its ability to predict teacher-generated labels (measured by AUC). The fine-tuned student often matched or exceeded the performance of the large teacher model directly.
• Topic Precision (AUPC): Using Area Under the Pareto Curve (AUPC) to measure the trade-off between the number of clusters and cluster purity, PRISM consistently outperformed Top2Vec and BERTopic across all datasets.
• Distillation Efficacy: PRISM trained with Range-Bound (RB) sampling and Embedding (Emb) data achieved the highest AUPC scores.
  • Example (IMDB): PRISM (RB + Emb) achieved an AUPC of 0.966, compared to 0.913 for Top2Vec (mpnet) and 0.957 for direct teacher clustering.
• Cost Efficiency: While training requires LLM queries (costing approx. $0.05–$0.20 per dataset), the inference phase is completely free of LLM costs, unlike LLM-guided clustering methods.

5. Significance

• Bridging the Gap: PRISM successfully bridges the gap between the high semantic resolution of frontier LLMs and the scalability requirements of real-world applications.
• Domain Adaptation: It provides a robust mechanism for adapting general-purpose embedding models to specific, narrow domains (e.g., disaster response, political fact-checking) where standard models fail to capture subtle distinctions.
• Practical Deployment: By eliminating the need for LLMs during inference, PRISM makes high-precision topic modeling feasible for researchers and practitioners with limited budgets or strict latency requirements.
• Nuance Preservation: The framework's ability to leave items unassigned (outliers) ensures that sensitive or unique narratives are not forced into incorrect categories, a critical feature for security and crisis monitoring.

In conclusion, PRISM demonstrates that careful knowledge distillation combined with strategic sampling allows small, local models to achieve state-of-the-art semantic resolution, offering a scalable alternative to expensive, LLM-dependent topic modeling pipelines.