# Classification: Why k-Nearest Neighbor method is more appropriate for a Mixture of Gaussians?

I'm reading a book named "The Elements of Statistical Learning" in which it states 2 scenarios when we are trying to predict the class label:

Scenario 1: The training data in each class were generated according to two bivariate Gaussian distribution with uncorrelated components and different means.

Scenario 2: The training data in each class came from a mixture of 10 low-variance Gaussian distributions, with individual means themselves distributed as Gaussian.

It's said that the Least Squares methods are more appropriate for Scenario 1, while k-nearest neighbor is more appropriate for Scenario 2. But I don't quite understand why.

Could anybody help explain the difference? Any help is greatly appreciated.

-
Ref: stanford.edu/~hastie/local.ftp/Springer/OLD//ESLII_print4.pdf , section 2.3. Have you understood the diagrams? –  leonbloy Jan 10 '13 at 19:52
Thanks for your reference book link! Do you mean the red-green graph? Yes, I guess so. I could see the difference in the predicted class. So, is it because Scenario 2 would be more messy that cannot be modelled by a global linear function? –  Tansy Jan 10 '13 at 20:26
Basically yes. But that reasoning could lead to conclude that k-NN is always better, it's important to understand why in Scenario 1 the linear discriminant is better. That's explained in the text. –  leonbloy Jan 10 '13 at 20:39
I could see a general explanation that least squares is smooth, quick, stable, and has lower variance. Did I miss any important explanation, like, mathematically speaking? Thanks a lot! –  Tansy Jan 10 '13 at 21:00