📊 statistics

A novel k-means clustering approach using two distance measures for Gaussian data

This paper proposes a novel k-means clustering algorithm for Gaussian data that utilizes both within-cluster and inter-cluster distance metrics alongside the Calinski-Harabasz criterion to achieve more robust convergence and improved outlier handling compared to traditional methods.

Original authors: Naitik Gada (Rochester Institute of Technology)

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Naitik Gada (Rochester Institute of Technology)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking into a massive, chaotic party where thousands of people are mingling, but no one knows who belongs with whom. There are no name tags, no group leaders, and no signs pointing to different tables. Your job is to figure out which people naturally hang out together. This is the world of unsupervised learning, a branch of computer science where algorithms try to find hidden patterns in messy data without being told the answers beforehand. One of the most popular tools for this job is called k-means clustering. Think of it like a game of musical chairs where the computer tries to group similar items together by finding a "center" for each group and pulling everyone closer to that center. The goal is to make sure everyone in a group is very similar to each other, while being very different from people in other groups. However, this game has a tricky flaw: the computer often gets stuck in a local "good enough" solution because it starts with a random guess about where the centers should be. If it picks the wrong starting spot, the whole grouping can go wrong. This matters because in the real world, from organizing customer data to analyzing medical images, getting these groups right is crucial for making smart decisions.

This paper introduces a new twist to the classic k-means game to make it more reliable. The author, Naitik H. Gada, suggests that the traditional method only looks at how close people are to their own group's center (called within-cluster distance). The new approach adds a second rule: it also checks how far apart the different groups are from each other (called inter-cluster distance). Imagine if, while grouping the partygoers, you didn't just ask, "Are you close to your friends?" but also, "Are you far enough away from the other tables?" By balancing these two measurements, the algorithm tries to create groups that are not only tight-knit but also clearly separated from one another.

The researchers tested this idea using two types of data. First, they created fake data sets that looked like neat, round clouds of points (simulating Gaussian data) with different levels of "messiness" or variance. They also tested the algorithm on real-world benchmark data sets, including the famous Iris flower data, a Wine chemical analysis, and a Breast Cancer medical data set. The results showed that the new method, which uses both distance measurements, consistently performed better than the traditional k-means. In the fake data tests, the new algorithm was more accurate and less likely to make mistakes when the data was messy or when the starting points were tricky. For example, on a 2D data set with high variance, the new method achieved an accuracy of 0.9801, compared to 0.9508 for the traditional method. On the Iris data set, it reached 0.8420 accuracy versus 0.7751 for the old way.

The paper also highlights that the new method is better at handling "outliers"—those party guests who are standing a bit far away from everyone else. In the Wine data set, the traditional method sometimes misclassified these distant points, while the new method correctly identified them. However, the authors are careful to note that while the new method is an improvement, it is not a magic bullet that solves every problem. It still struggles a bit with the initial starting positions of the groups, and its performance on very high-dimensional data (like the 9-dimensional Breast Cancer set) was only slightly better than the traditional method. The study suggests that adding this second distance measure makes the clustering "solidified and more robust," but it remains a work in progress that opens the door for even more sophisticated research in the future.

Technical Summary: A Novel k-means Clustering Approach Using Two Distance Measures for Gaussian Data

Problem Statement
Clustering analysis is a fundamental unsupervised learning technique used to identify homogeneity within unstructured, unlabelled data. Among various methods, k-means clustering is widely utilized due to its computational efficiency compared to hierarchical approaches. However, traditional k-means suffers from two primary limitations:

Sensitivity to Initialization: The algorithm is highly sensitive to the initial positions of cluster centers, often causing convergence to local minima rather than the global optimum.
Single Metric Optimization: The standard algorithm minimizes only the Within-Cluster Distance (WCD) to ensure homogeneity within groups. It does not explicitly maximize the separation between different groups (Inter-Cluster Distance or ICD).
Dependence on Pre-determined $k$ : The number of clusters ( $k$ ) must be specified beforehand, which is difficult for high-dimensional data where visualization is impractical.

Existing literature has attempted to address these issues through various initialization methods (e.g., genetic algorithms, simulated annealing) and alternative distance measures (e.g., Manhattan, Chebyshev). However, many studies rely heavily on synthetic data, lack rigorous quantification using metrics like F1-score or sensitivity, or fail to incorporate a distance metric specifically designed to maximize inter-cluster separation.

Methodology
The paper proposes a novel modification to the standard k-means algorithm that incorporates both Within-Cluster Distance (WCD) and Inter-Cluster Distance (ICD) as distance metrics.

Cluster Determination: The number of clusters ( $k$ ) is not arbitrarily chosen but determined using the Calinski-Harabasz index, with the Elbow method used for validation.
Algorithm Logic:
- Initialization: Cluster centers are initialized via a random permutation of indices from the dataset.
- Dual-Metric Optimization: Unlike traditional k-means which minimizes the sum of squared Euclidean distances from points to their centroids (WCD), the proposed algorithm optimizes a relative cost function. This function is defined as the ratio of the total WCD to the total ICD.
- WCD Definition: The sum of squared distances from a cluster center to all points within that cluster.
- ICD Definition: The sum of squared distances between a point in cluster $i$ and all points not in cluster $i$ , summed over all clusters.
- Iteration: The algorithm iteratively updates centroids by minimizing this relative cost. The process continues until centroids stabilize.
Evaluation Metrics: To quantify performance, the study utilizes a Confusion Matrix to calculate Overall Accuracy (OA), Precision (Positive Predictive Value), Recall (Sensitivity), and the F1 Score.

Experimental Setup
The algorithm was evaluated on two categories of datasets:

Synthetic Data: Four datasets generated from normal distributions with varying dimensions (2D and 3D) and variances (0.5 and 1.0). These were designed to test performance on both well-defined and overlapping (ill-defined) clusters.
Benchmark Data: Three datasets from the UCI Machine Learning Repository:
- Iris: 150 instances, 4 features, 3 classes.
- Wine: 178 instances, 13 chemical attributes, 3 classes.
- Breast Cancer (Wisconsin): 569 instances, 9 features, 2 classes (Benign/Malignant).

All experiments were run for 100 iterations with identical random initialization seeds for both the proposed and traditional k-means algorithms to ensure a fair comparison.

Key Results
The proposed algorithm consistently outperformed the traditional k-means across all datasets, though the magnitude of improvement varied:

Synthetic Data:
- In 2D data with variance 0.5, the proposed method achieved 97.05% accuracy versus 96.07% for traditional k-means.
- In 2D data with variance 1.0 (more overlapping clusters), the gap widened to 98.01% vs. 95.08%.
- In 3D data, the proposed method showed significant robustness, particularly in variance 0.5 scenarios where traditional k-means accuracy dropped to ~60% in certain iterations due to poor initialization, while the proposed method remained near 100%.
Benchmark Data:
- Iris: Proposed accuracy 84.20% vs. 77.51% (traditional). The proposed method also correctly identified specific outliers that the traditional method misclassified.
- Wine: Proposed accuracy 75.16% vs. 66.37%. The proposed method demonstrated significantly lower volatility across iterations.
- Breast Cancer: The difference was minimal (96.12% vs. 95.82%), but the proposed method remained slightly more consistent and less volatile.

Significance and Claims
The paper claims that the inclusion of the Inter-Cluster Distance (ICD) metric alongside the standard WCD creates a more robust clustering algorithm. The primary contributions are:

Improved Robustness: The dual-metric approach reduces sensitivity to initial cluster center positions, resulting in more consistent convergence and fewer instances of the algorithm getting stuck in poor local minima.
Outlier Handling: The proposed method demonstrates a superior ability to correctly classify outliers into their true clusters compared to the traditional method.
Quantifiable Validation: Unlike many prior studies that rely solely on synthetic data or qualitative assessments, this work validates the algorithm on real-world benchmark datasets using a comprehensive set of metrics (Accuracy, Precision, Recall, F1).
High-Dimensional Viability: Although the algorithm was conceptualized for lower-dimensional Gaussian data, the stress test on the 9-dimensional Breast Cancer dataset suggests the approach is robust enough for higher-dimensional applications.

The authors conclude that while the accuracy gains may be modest in some high-dimensional cases, the reduction in volatility and the improved handling of ambiguous data structures validate the hypothesis that optimizing for both intra-cluster homogeneity and inter-cluster heterogeneity yields a superior clustering solution. The work sets a precedent for further research into nuanced numerical methods to address k-means initialization sensitivity.

More like this