LUMI: Unsupervised Intent Clustering with Multiple Pseudo-Labels

1. Problem Statement

The paper addresses the challenge of unsupervised intent clustering in conversational search systems. The goal is to group unlabeled user utterances into clusters of similar intents without access to ground-truth labels or a pre-defined number of clusters ($K$).

Limitations of Current Approaches:
Existing methods often rely on Large Language Models (LLMs) to generate pseudo-labels or identify similar text pairs. However, they suffer from two main issues:

1. Single-Label Instability: Current approaches assign only a single pseudo-label per text. Refining text representations toward a single, potentially noisy label can be unstable.
2. Binary Similarity: Text similarity is often treated as a binary decision (similar or not), failing to account for continuous degrees of similarity.
3. Dependency on $K$: Many state-of-the-art (SOTA) methods require knowing the number of clusters ($K$) during the embedding refinement phase, which is impractical in real-world scenarios where $K$ is unknown.

2. Methodology: LUMI

LUMI (Unsupervised Intent Clustering with Multiple Pseudo-Labels) is a training-free, label-free method that enhances text representations by leveraging multiple pseudo-labels. The method operates in three main stages:

A. Pseudo-Label Generation

1. Initial Labeling: For each text $x_i$, an LLM is prompted to generate a single initial pseudo-label. To ensure semantic relevance, the LLM is provided with the top-$b$ (e.g., $b=10$) nearest neighbors of $x_i$ based on a pre-trained embedder.
2. Multi-Label Classification: The LLM is then prompted to perform multi-label classification. It is given the text and its top-$b$ neighbors (each with a different initial pseudo-label) and asked to select a set of relevant pseudo-labels ($O_i$) for the text.
3. Final Label Set: The final pseudo-label set $L_i$ for text $x_i$ consists of the initial label plus the selected labels from the multi-label step.

B. Label-Level Pooling

To create a stable representation for each text:

1. Averaging: The embeddings of all pseudo-labels in the set $L_i$ are averaged to create a single "averaged pseudo-label embedding" ($\bar{z}$).
   • Theoretical Justification: The authors prove that averaging multiple pseudo-labels reduces the variance (noise) of the label generation process compared to using a single label, bringing the representation closer to the true cluster centroid.
2. Text-Label Fusion: The final refined embedding ($z'$) is computed by averaging the original text embedding and the averaged pseudo-label embedding:
   $$z' = \frac{f(x) + \bar{z}}{2}$$

C. Text-Level Pooling

To further refine representations based on continuous similarity:

1. Similarity Weighting: Instead of binary selection, LUMI uses Jaccard similarity between the pseudo-label sets of two texts ($L_i$ and $L_j$) to determine their similarity weight ($w_{ij}$):
   $$w_{ij} = \frac{|L_i \cap L_j|}{|L_i \cup L_j|}$$
2. Weighted Aggregation: The final refined embedding ($\tilde{z}'$) for a text is a weighted average of its own embedding and the embeddings of similar texts, weighted by their shared label extent.
   $$\tilde{z}'_i = \frac{\sum_{x_j \in D} w_{ij} z'_j}{\sum_{x_j \in D} w_{ij}}$$

3. Key Contributions

1. Theoretical Justification: The paper provides a mathematical proof demonstrating that averaging multiple pseudo-labels reduces noise and variance, theoretically guaranteeing a representation closer to the cluster centroid than single-label approaches.
2. Novel Hybrid Approach: LUMI bridges the gap between "LLM-as-a-matcher" (selecting similar candidates) and "LLM-as-a-labeler" (generating labels) by utilizing multiple pseudo-labels for both representation enrichment and similarity weighting.
3. Training-Free & $K$-Free: The method requires no labeled training data and does not need the number of clusters ($K$) to be known during the embedding refinement stage, making it highly practical for real-world deployment.
4. Robustness: The approach is shown to be stable across different LLMs (Gemma, Llama) and embedders (MiniLM, Instructor).

4. Experimental Results

The authors evaluated LUMI on four benchmark datasets: Bank77, CLINC150, Mtop, and Massive.

• Performance: LUMI consistently outperformed SOTA baselines (including IDAS, SPILL, ClusterLLM, and LLMEdgeRefine) across most datasets and configurations.
  • It achieved higher Normalized Mutual Information (NMI) and Clustering Accuracy (Acc) scores.
  • On average, it improved accuracy by 10–13 percentage points over GPT-3.5 based methods and 0–3 points over GPT-4 based methods, despite using smaller, open-source LLMs (Gemma/Llama).
• Ablation Studies:
  • Adding Label-Level Pooling significantly improved performance over pre-trained embeddings.
  • Adding Text-Level Pooling (using Jaccard weights) provided further gains on three of the four datasets.
• Robustness: The method showed low sensitivity to the candidate count parameter ($b$), maintaining stable performance across variations.
• Error Analysis: The only dataset where LUMI did not strictly dominate was Bank77. The authors attribute this to the fine-grained nature of banking intents (e.g., distinguishing between "pending top-up" and "automatic top-up"), where the LLM-generated pseudo-labels were sometimes too broad (noisy), covering multiple distinct ground-truth intents.

5. Significance

LUMI represents a significant advancement in unsupervised short-text clustering by:

• Eliminating the need for ground-truth $K$: This removes a major bottleneck in deploying clustering systems for dynamic conversational agents.
• Reducing reliance on massive models: It achieves SOTA performance using smaller, open-source LLMs, making it more cost-effective and accessible.
• Improving Stability: By leveraging multiple pseudo-labels and continuous similarity weighting, it mitigates the noise inherent in single-shot LLM labeling, providing a more robust foundation for intent discovery.

The source code is publicly available, facilitating reproducibility and further research in unsupervised intent clustering.