Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I'm reading a book named "The Elements of Statistical Learning" in which it states 2 scenarios when we are trying to predict the class label:

Scenario 1: The training data in each class were generated according to two bivariate Gaussian distribution with uncorrelated components and different means.

Scenario 2: The training data in each class came from a mixture of 10 low-variance Gaussian distributions, with individual means themselves distributed as Gaussian.

It's said that the Least Squares methods are more appropriate for Scenario 1, while k-nearest neighbor is more appropriate for Scenario 2. But I don't quite understand why.

Could anybody help explain the difference? Any help is greatly appreciated.

share|improve this question
    
Ref: stanford.edu/~hastie/local.ftp/Springer/OLD//ESLII_print4.pdf , section 2.3. Have you understood the diagrams? –  leonbloy Jan 10 '13 at 19:52
    
Thanks for your reference book link! Do you mean the red-green graph? Yes, I guess so. I could see the difference in the predicted class. So, is it because Scenario 2 would be more messy that cannot be modelled by a global linear function? –  Tansy Jan 10 '13 at 20:26
    
Basically yes. But that reasoning could lead to conclude that k-NN is always better, it's important to understand why in Scenario 1 the linear discriminant is better. That's explained in the text. –  leonbloy Jan 10 '13 at 20:39
    
I could see a general explanation that least squares is smooth, quick, stable, and has lower variance. Did I miss any important explanation, like, mathematically speaking? Thanks a lot! –  Tansy Jan 10 '13 at 21:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.