4
$\begingroup$

In reading the book "An Introduction to Statistical Learning with Applications in R", I came across this graph:

enter image description here

It shows that the point-wise variance is larger at the ends of the regression curve. Why is that? I thought that the variance may be larger because there seem to be fewer data points near the end, but the variance is calculated on the coefficients so the number of data points used in the estimate has no impact. Then I thought that the variance is larger because the X values are larger, but the graph on the left side shows slight increases in variance on both sides (even when X is small).

In general, I have read that polynomials have notorious end behaviours - what causes this?

Thanks.

$\endgroup$
  • $\begingroup$ If you don't get an satyisfaying answer try asking in stats-stackexchange.com $\endgroup$ – Manuel Aug 28 '15 at 19:04
1
$\begingroup$

It's essentially due to Runge's Phenomenon for interpolating polynomials. Even without uncertainty, the behavior of higher order polynomials at their endpoints is very sensitive to the parameter values.

Here is a link to a technical explanation that I will regurgitate here. It relates Runge's Phenomenon to regression. Below is a intuitive, but admittedly non-rigorous explanation.

A good way to think about it is that, in order to be a good fit to the sample data (i.e., not veer completely out of the range of the data), you need to balance out the contributions from the higher order terms using lower order terms. An imbalance in this over the range of the function will result in a bad fit, so least squares requires ever more stringent "tuning" as you add more polynomial terms. Thus, small errors result in the endpoints "wagging" a lot more than the center of the function. This is especially bad if you don't have a lot of data at the edges of your interval.

This is not totally rigorous, but hopefully it shows intuitively why this would be so. The details would come from actually calculating the CI bounds, but I don't think that is as informative for the intuition.

$\endgroup$
  • 1
    $\begingroup$ I don't think you are answering the OP's question. First of all, regression is not interpolation. In interpolation, we try to find a function (usually a polynomial) that approximates another function. In regression, however, there may be data points with identical abscissa (age) but different ordinates (wages) and hence the data set is not presented as a sample of some function. $\endgroup$ – user1551 Aug 29 '15 at 8:24
  • $\begingroup$ Secondly, the OP asks why the deviation of the data points from the interpolating curve are larger near the end of the curve, but in fact this is data dependent. E.g. if the data points are concentrated at (20,50) and (80,50) and there are a few data points with age between 30 and 70 but large variance in wages, the regression curve will likely pass through somewhere near (20,50) and (80,50) and the phenomenon mentioned by the OP may not occur. $\endgroup$ – user1551 Aug 29 '15 at 8:29
  • $\begingroup$ @user1551 I disagree with your second comment. Op is referring to the CI bands in the figures. Second, I know this is not interpolation...however the underlying reason that higher order polynomial are erratic is related to this phenomenon. $\endgroup$ – user237392 Aug 29 '15 at 9:17
  • $\begingroup$ I also think you haven't answered why the variance increases at the edges. $\endgroup$ – Nameless Aug 30 '15 at 14:55
  • $\begingroup$ @Nameless this phenomenon is not unique to higher order polynomials. It even happens in linear regression too. The underlying math gets messy beyond the linear case. What additional type of information are you looking for? $\endgroup$ – user237392 Aug 30 '15 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.