I've read the Wikipedia article and a lot of posts on stackexchange (like this really thorough one) on determining the number of clusters in a data set. Based on that, I am currently using the silhouette analysis in MATLAB.

Clustering $2$-dimensional data (around $10^3$ points) works fine, I can determine the average silhouette value for $k=2,3,\ldots,k_{\mathrm{max}}$ for some $k_{\mathrm{max}}\in\mathbb{N}$ and of those $k$, pick the one that corresponds to the highest average silhouette value. That takes less than a minute to run.

However, with $4$-dimensional data (around $10^5$ points), this approach takes a long time. The clustering itself (using kmeans in MATLAB) is still fairly quick, but calculating the silhouette value is slow. So my thought was: perhaps one of the other methods is faster. Hence my question:

Can anyone provide insight into the performance in higher dimensions of the different methods for choosing the optimal number of clusters in $k$-means clustering?

  • $\begingroup$ @user1952009. I wouldn't know why it wouldn't be slow and hard. Neither have I claimed that. Neither does my question doubt that. I asked for a comparison between different methods. They might all be slow, but perhaps one of them is less slow than the others. $\endgroup$ – Eric S. Mar 25 '16 at 9:23
  • $\begingroup$ you already have a comparison of many methods on the link you referred to. if you don't know them well, start by understanding each one. if you know them already, be more specific. $\endgroup$ – reuns Mar 25 '16 at 10:23
  • $\begingroup$ @user1952009. Why must I be more specific? I understand the methods and could program them all into MATLAB and do multiple comparisons for different sample sizes in different dimensions. But that is a lot of work, for which I don't have the time right now. Furthermore, that is only empirical and perhaps someone knows a more elegant/analytical approach. Your comments are not really helpful or constructive by the way, so perhaps don't bother... $\endgroup$ – Eric S. Mar 25 '16 at 10:41
  • $\begingroup$ for example, what are you data points ? do they follow some generative model ? how many clusters ? how many points per clusters ? did you study other methods of clustering (for example hierarchical clustering, EM gaussian mixtures, kernel k-means, spectral clustering) ? what your clustering is then useful for ? etc. we are talking of some problems which are never perfectly solved in the general case, and you are asking about the general case. so my advice is to study a little more, and then to come back and be more specific. $\endgroup$ – reuns Mar 25 '16 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.