Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm reading a book named "The Elements of Statistical Learning" in which it states 2 scenarios when we are trying to predict the class label:

Scenario 1: The training data in each class were generated according to two bivariate Gaussian distribution with uncorrelated components and different means.

Scenario 2: The training data in each class came from a mixture of 10 low-variance Gaussian distributions, with individual means themselves distributed as Gaussian.

It's said that the Least Squares methods are more appropriate for Scenario 1, while k-nearest neighbor is more appropriate for Scenario 2. But I don't quite understand why.

Could anybody help explain the difference? Any help is greatly appreciated.

share|cite|improve this question
Ref: , section 2.3. Have you understood the diagrams? – leonbloy Jan 10 '13 at 19:52
Thanks for your reference book link! Do you mean the red-green graph? Yes, I guess so. I could see the difference in the predicted class. So, is it because Scenario 2 would be more messy that cannot be modelled by a global linear function? – Tansy Jan 10 '13 at 20:26
Basically yes. But that reasoning could lead to conclude that k-NN is always better, it's important to understand why in Scenario 1 the linear discriminant is better. That's explained in the text. – leonbloy Jan 10 '13 at 20:39
I could see a general explanation that least squares is smooth, quick, stable, and has lower variance. Did I miss any important explanation, like, mathematically speaking? Thanks a lot! – Tansy Jan 10 '13 at 21:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.