Technical Summary: CLUBench: A Clustering Benchmark

Problem Statement

Clustering is a fundamental data science task with a long history of algorithmic development, ranging from conventional methods to deep learning-based approaches. Despite this progress, the field lacks a systematic, large-scale empirical evaluation that jointly considers conventional algorithms, deep clustering (DC) methods, and emerging foundation model (FM)-based strategies. Existing surveys often rely on qualitative comparisons or limited datasets, leading to biased conclusions and a lack of guidance for algorithm selection and deployment. Furthermore, there is no unified, easy-to-use toolbox to support reproducible benchmarking across these diverse paradigms.

Methodology

The authors introduce CLUBench, a comprehensive benchmark designed to address these gaps through rigorous experimental evaluation and analysis.

• Scope and Scale: The benchmark evaluates 24 clustering algorithms across 131 diverse datasets covering three modalities: tabular, text, and image. This results in 178,815 experiments.
• Algorithm Categories:
  • Conventional Clustering (CC): 14 algorithms spanning six categories: partition-based (e.g., KMeans, KernelKMeans), hierarchical (e.g., AggClu, BIRCH), density-based (e.g., DBSCAN, MeanShift), model-based (GMM), subspace-based (e.g., SSC, LRR, k-FSC), and graph-based (e.g., SpeClu, AutoSC).
  • Deep Clustering (DC): 10 state-of-the-art methods (2016–2025) categorized by technical paradigm: autoencoder-based (DEC, IDEC), deep subspace (DSCN, EDESC), contrastive learning (ConClu, DMICC), information theory (PICA, DIVC), and self-supervision (P2OT, LFSS).
• Experimental Protocol:
  • Hyperparameter Tuning: Algorithms are evaluated under both default settings and best-tuned hyperparameter configurations (HPCs) to assess robustness and performance ceilings.
  • Foundation Models: The study incorporates FM-based strategies, comparing top-performing baselines operating on pretrained embeddings (ResNet, CLIP, BERT, Llama3, OpenAI) against specialized deep clustering methods. For tabular data, in-context learning with LLMs is explored.
  • Metrics: Performance is measured using Clustering Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI).
• Analytical Framework:
  • Similarity Analysis: Performance vectors are constructed for algorithms and datasets to analyze similarities using t-SNE.
  • Low-Rank Analysis: Singular Value Decomposition (SVD) is applied to performance matrices to investigate low-rank structures and the feasibility of matrix completion for performance prediction.
  • Model Selection: Regression models (XGBoost, LightGBM, Random Forest) are trained to map dataset meta-features to performance matrices, aiming to automate algorithm and hyperparameter selection.
• Toolbox: A unified Python toolbox, compatible with scikit-learn, encapsulates the source code for all evaluated methods, providing a consistent interface for deployment and extension.

Key Results

• Conventional vs. Deep Clustering: Contrary to the expectation that deep learning offers superior performance, the study finds that all evaluated deep clustering methods do not exhibit a significant advantage over top-performing conventional algorithms (specifically KMeans and Spectral Clustering/SpeClu) in terms of average performance across the 131 datasets.
• Impact of Hyperparameters: There is a substantial performance gap between default and best-tuned configurations for both CC and DC methods, highlighting the high sensitivity of clustering algorithms to hyperparameter selection.
• Foundation Model Integration:
  • Image & Text: Combining pretrained embeddings (e.g., from CLIP, ResNet, LLMs) with conventional algorithms (KMeans, SpeClu) yields effective and efficient clustering, often rivaling or surpassing specialized deep clustering methods.
  • Tabular Data: Existing tabular foundation models show limitations in unsupervised scenarios. While LLMs via in-context learning show promise on small datasets, they do not consistently outperform traditional methods like SpeClu on average.
• Algorithm and Dataset Similarity: Algorithms with similar methodological designs (e.g., subspace methods) cluster together in performance space, suggesting that testing a few representative methods per cluster can approximate the broader performance landscape.
• Low-Rank Structure: Performance matrices exhibit a strong low-rank structure (cumulative contribution ratio >90% for the first 60 singular values). This enables reliable prediction of overall performance from incomplete observations via matrix completion.
• Model Selection Feasibility: Regression-based model selection strategies can effectively approximate the relationship between dataset meta-features and clustering performance, achieving results comparable to or surpassing specific baselines.

Significance and Claims

The paper claims that CLUBench provides a challenging and challenging benchmark that reveals the intrinsic difficulty of clustering diverse real-world data, even in the era of dominant foundation models. Its primary contributions are:

1. Empirical Evidence: It offers a quantitative evaluation demonstrating that, for general clustering tasks, carefully tuned conventional algorithms often provide a higher performance ceiling than deep learning methods, particularly on tabular data.
2. Practical Guidance: It identifies that for image and text tasks, the combination of pretrained embeddings with conventional clustering is a strong, efficient baseline.
3. Research Direction: It highlights the need for future research to focus on automatic model selection, the design of deep clustering frameworks tailored specifically for tabular data, and the careful accounting of potential data leakage in foundation model evaluations.
4. Resource Availability: By providing a unified toolbox and open-sourcing all datasets and results, the authors aim to facilitate reproducible research and practical deployment in the clustering community.

The authors maintain a modest stance, noting that while their findings offer valuable insights, clustering remains a nontrivial problem, and the proposed benchmark serves as a foundation for further investigation rather than a final solution to algorithm selection.