# Theoretical basis for overfitting

There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions over complex ones in the general case, and that it had to do with information theory.

Does anyone know what he was referring to?

As an example: consider least square's. Obviously we could find a polynomial of very high degree which has zero error, but we prefer a linear equation with higher error. Why should this be?

(I am familiar with some basic notions like entropy, but not much more than that, so simpler explanations would be much preferred. Although I understand that if it's complex, it's complex.)

-
For doing what? – Qiaochu Yuan Dec 21 '10 at 0:51
@Qiaochu: The specific example was interpolation by a polynomial, and why splines are preferred to interpolating at a large number of points. I understood him to mean it was a general phenomenon though. – Xodarap Dec 21 '10 at 0:55
The problem with high-degree polynomials is that they tend to wiggle. A lot. – J. M. Dec 21 '10 at 1:08
Also, the adage "high order does not imply high accuracy" applies as well. – J. M. Dec 21 '10 at 1:08
This might be helpful. isites.harvard.edu/fs/docs/icb.topic539621.files/lec7.pdf – user17762 Dec 21 '10 at 1:09