📊 statistics

SPINEX-Clustering: Similarity-based Predictions with Explainable Neighbors Exploration for Clustering Problems

This paper introduces SPINEX-Clustering, a novel similarity-based algorithm that leverages higher-order interactions across subspaces to achieve top-tier performance and explainability across 51 diverse datasets while maintaining moderate computational complexity compared to 13 established clustering methods.

Original authors: MZ Naser, Ahmed Naser

Published 2026-08-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: MZ Naser, Ahmed Naser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast world of modern data, information often arrives as a chaotic cloud of points, each representing a person, a sensor reading, or a biological sample. To make sense of this noise, scientists use a technique called clustering, which acts like a sorting mechanism to group similar items together while keeping different items apart. The goal is to find hidden patterns where objects within a group share more in common with each other than with those outside the group. For decades, researchers have relied on established methods to perform this sorting, but these traditional tools often struggle when the data is messy, high-dimensional, or shaped in complex, irregular ways. They frequently assume that groups are simple, round shapes or require the user to guess the number of groups beforehand, which is not always possible in real-world scenarios. As datasets grow larger and more intricate, the need for a more flexible and intelligent way to organize information has become critical.

A new approach called SPINEX, developed by researchers at Clemson University and the University of Manitoba, offers a fresh perspective on this sorting challenge. Rather than relying on a single rigid rule, SPINEX acts as a versatile explorer that examines data through multiple lenses. It looks at how closely data points resemble one another using various mathematical measures of similarity, such as how their values rise and fall together or how they align in space. Crucially, the algorithm is designed with flexibility, allowing it to work with or without a predefined number of clusters; it can autonomously determine an appropriate number of groups based on the data's structure or operate within user-specified constraints. It investigates the neighborhood of each point, understanding how local connections form larger structures. This allows it to discover clusters of any shape, whether they are tight spheres, winding spirals, or scattered clouds. Furthermore, unlike many "black box" algorithms that provide an answer without explanation, SPINEX is designed to be transparent. It can show exactly why a specific data point was placed in a certain group, detailing which features contributed most to that decision, making the results understandable and trustworthy for human users.

To test if this new method truly works, the researchers put SPINEX through a rigorous series of trials against thirteen other well-known clustering algorithms. They ran these tests on fifty-one different datasets, ranging from computer-generated simulations designed to mimic difficult scenarios to real-world data from various scientific fields. The performance was measured using several standard criteria that check how well the groups are separated from each other and how consistent the members within each group are. The results showed that while the standard SPINEX algorithm ranked last (17th out of 17) on synthetic data, its specialized variants consistently ranked among the top performers. In fact, several versions of the new algorithm, which incorporated techniques like dimensionality reduction or multi-level clustering, placed within the top five best-performing methods across the board. One variant, which incorporated a technique to simplify the data before sorting, tied for second place overall, demonstrating a strong ability to handle complex structures. While the algorithm showed moderate computational complexity, meaning it is efficient enough for large datasets, its greatest strength appeared to be its adaptability. It performed well across diverse conditions, proving that its strategy of combining multiple similarity measures with neighbor exploration is effective.

The study also highlighted a significant advantage in how the algorithm handles the "why" behind its decisions. By analyzing the contribution of individual features to the similarity between points, SPINEX can explain its logic. For instance, it can identify that two data points were grouped together primarily because they shared a specific pattern in their values, rather than just being generally close. This explainability is a vital feature for fields where understanding the reasoning behind a classification is just as important as the classification itself. The researchers found that while some older algorithms excelled at specific types of data, they often struggled with others, whereas the optimized SPINEX variants maintained a high level of performance across the board. The findings suggest that this new method provides a robust and flexible tool for organizing complex information, offering a balance of accuracy, efficiency, and clarity that existing tools often lack. As data continues to grow in volume and complexity, approaches that can not only find patterns but also explain them will become increasingly essential for turning raw information into meaningful insight.

Technical Summary of SPINEX-Clustering

Problem Statement
Clustering remains a fundamental challenge in unsupervised machine learning, particularly as datasets grow in size, dimensionality, and complexity. Traditional algorithms often rely on rigid assumptions, such as isotropic and equidistant clusters (e.g., K-Means) or fixed parameter settings that struggle with varying densities and arbitrary shapes (e.g., DBSCAN). Furthermore, many existing methods lack flexibility in determining the optimal number of clusters, suffer from scalability issues in high-dimensional spaces, and offer limited explainability regarding how clusters are formed. The paper identifies a need for an algorithm that can adapt to intrinsic data properties, handle noise and outliers, scale to large datasets, and provide interpretable insights into cluster formation.

Methodology: The SPINEX Algorithm
The paper introduces SPINEX (Similarity-based Predictions with Explainable Neighbors Exploration), a novel clustering algorithm designed to address the aforementioned limitations. SPINEX operates on the principle that similarity and higher-order interactions across multiple subspaces are key to forming coherent groups. Its technical architecture includes:

Multi-Strategy Similarity: Unlike algorithms relying on a single metric, SPINEX employs a diverse set of similarity measures, including Pearson correlation, Spearman rank correlation, kernel-based similarity (RBF), and cosine similarity. It computes these matrices to capture different aspects of data relationships.
Adaptive Thresholding: The algorithm dynamically determines cluster membership thresholds based on the statistical properties of the similarity matrix (e.g., median plus standard deviation) or user-defined percentiles, reducing the need for manual parameter tuning.
Multi-Level and Hierarchical Clustering: SPINEX supports hierarchical structures, allowing for the identification of both broad groups and nuanced sub-groups. It utilizes an optimized merging method that iteratively combines clusters based on average similarity exceeding a dynamic threshold.
Scalability and Optimization: To handle large datasets, SPINEX incorporates dimensionality reduction techniques (PCA, t-SNE, UMAP) and approximation methods (random sampling). It also leverages parallel processing (via ProcessPoolExecutor) for computationally intensive tasks like similarity matrix generation and metric calculation.
Explainability: A core feature of SPINEX is its ability to provide explainable insights. It includes modules for similarity contribution analysis (determining which features drive the similarity between observations) and nearest neighbor analysis (identifying the $k$ -nearest neighbors and the specific feature contributions that define these relationships).
Self-Evaluation: The algorithm evaluates its own performance using internal metrics (Silhouette Score, Calinski-Harabasz Index, Davies-Bouldin Index) and, when ground truth is available, external metrics (Homogeneity, Completeness, V-Measure) to select the best-performing similarity method.

Benchmarking and Results
The authors conducted a rigorous benchmarking study comparing SPINEX against 13 established clustering algorithms: Affinity Propagation, Agglomerative, Birch, DBSCAN, Gaussian Mixture, HDBSCAN, K-Means, KMedoids, Mean Shift, MiniBatch K-Means, OPTICS, Spectral Clustering, and Ward Hierarchical.

Datasets: The evaluation spanned 51 datasets, comprising 33 synthetic datasets (simulating various conditions like non-spherical shapes, varying densities, and noise) and 18 real-world datasets from diverse domains.
Performance:
- On synthetic data, HDBSCAN achieved the highest overall ranking. However, specific SPINEX variants incorporating dimensionality reduction (denoted as SPINEX_T, SPINEX_T_PCA, SPINEX_T_TSNE, and SPINEX_T_UMAP) consistently ranked within the top 5, demonstrating strong performance in Homogeneity and Completeness.
- The standard SPINEX algorithm (without these specific optimizations) ranked 17th (last) in the synthetic tests. The authors note this highlights the necessity of specialized variants for optimal performance.
- The provided text outlines the setup for the 18 real-world datasets but does not present the specific performance results for these datasets.
Complexity: The paper presents a complexity analysis indicating that SPINEX maintains moderate computational complexity, aided by its approximation and parallel processing capabilities.

Key Contributions and Significance
The paper claims the following contributions:

Novel Algorithmic Framework: The introduction of SPINEX, which integrates multiple similarity measures, adaptive thresholding, and multi-level clustering into a unified framework.
Explainability: The provision of built-in mechanisms to explain clustering decisions by analyzing feature contributions and neighbor relationships, addressing the "black box" nature of many clustering methods.
Comprehensive Benchmarking: A thorough evaluation across 51 datasets against 13 competitors, providing empirical evidence of specific SPINEX variants' ability to rank within the top-performing algorithms on synthetic data.
Flexibility: The algorithm's ability to operate with or without a predefined number of clusters and its adaptability to various data types and distributions.

The authors conclude that SPINEX offers a robust alternative to traditional clustering methods, particularly in scenarios requiring interpretability and adaptability to complex data structures. They acknowledge that while the foundational concept is sound, the integration of dimensionality reduction and specific optimizations (as seen in the top-ranked variants) is essential for achieving state-of-the-art performance. Future research directions are noted as necessary to further refine these capabilities.

Technical Summary of SPINEX-Clustering

More like this