1
$\begingroup$

I want to find number of cluster in the real world data set. So, I validate the spectral clustering by using some indexes as shown in figures below? But as you seen in figures the results are very different from one index to other. How to determine the good number of clusters in the spectral clustering method? If u don't have complete answer give me any suggestion or idea that may help me.

Thanks
enter image description here enter image description here enter image description here enter image description here enter image description here enter image description here

$\endgroup$
1
$\begingroup$

I believe there exists no best method to assess clustering quality, independently of the data. You can check this answer for many other ways to assess such quality, in the case of k-means. Thiscan be good starting point: the relationship between spectral clustering and a suitable generalization of k-means is contained here .An important discrimination in quality assessment is between internal and external methods: internal ones check the dependence of an objective function on the number of clusters (for example, this is what happens with the "elbow" method). External criteria refer to an optimal "choice", which has to be known a priori, and try to compare the optimum with the computed clustering. The optimal choice has to be inferred looking at the data, knowing the problem in detail, or by other methods. In your specific case, different methods seem to give quite different answers. I would try now to visualize the clusters, and to find and select the choice which gives the most reasonable classification of the data themselves.

$\endgroup$
  • $\begingroup$ Thanks for answer. I agree with your description but as you mentioned I give quite different answers and don't know how to conclude from them and I should mention that my data is about 5k*5k and it is very difficult to visualize and analyse the data. $\endgroup$ – Fatime Aug 26 '13 at 9:11
  • 1
    $\begingroup$ you are welcome. Please, feel free to flag the answer if you found it useful. Returning to your analysis: I have noticed you did not include the silhouette coefficient: is there any reason for that? More than the sheer number of data, I would have a look at the origin of them, their structure and so on. Try to select 2 features and make a plot of the clusters. $\endgroup$ – Avitus Aug 26 '13 at 9:19
  • 1
    $\begingroup$ In many cluster packages you can compute the silhouette coefficient of the obtained clustering. I was just wondering whether you excluded it from the above list of indices due to theoretical reasons. $\endgroup$ – Avitus Aug 26 '13 at 9:33
  • 1
    $\begingroup$ I found also this answer for you :-) It is quite interesting. I hope it helps stats.stackexchange.com/questions/65712/… $\endgroup$ – Avitus Aug 26 '13 at 9:39
  • 1
    $\begingroup$ Ooops...my bad: I did not recognize the silhoutte at a first glance! Sparsity can be a critical feature of data...see for example the answers in stats.stackexchange.com/questions/10122/… $\endgroup$ – Avitus Aug 26 '13 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.