# Difference between best predictor and best linear predictor

Consider $Y=sin(X)+\epsilon$, where $X \sim U(0,2\pi)$ and $\epsilon \sim N(0, \sigma ^2)$. Then, I think that the best predictor would be $E[Y|X]=E[sin(X)+\epsilon|X]=E[sin(X)|X]+E[\epsilon|X]=E[sin(X)]=0$. (However I am not sure on the last step, I used $E[sin(X)]=\int_0^{2\pi} \frac{sin(x)}{2\pi}$.)

Since $E[Y|X]$ is not linear, then we would have to find the best linear predictor. I think to find the best linear predictor we would use $Y=E[Y]+\frac{ Cov(X,Y)}{Var(X)}(X-E[X])$. But I wanted to make sure before I started solving for the parameters. I got that $E[X]=\pi$, $E[Y]=0$, but I cannot find Cov(X,Y). If $E[Y|X]$ truly does equal 0 then I think Cov(X,Y) will also equal 0.

So I get that both the best predictor and linear predictor equal 0, and I don't think that is correct.

It is difficult for me to learn without being given examples so I am sorry in advanced for my many mistakes, thank you for your help.

## 1 Answer

Consider the example here. For $X$ and $\epsilon\sim i.i.d. N(0,1)$, and $Y=X^2+\epsilon$, the best predictor for $Y$ without knowing $\epsilon$ is $X^2$. What is the best linear predictor $a+bX$? You find the values $a,b$ that minimize the prediction error

\begin{align} \langle(a+bX-Y)^2\rangle&=\langle(a+bX-X^2-\epsilon)^2\rangle=\langle(a+bX-X^2)^2\rangle+\langle\epsilon^2\rangle\\ &=a^2+(b^2-2a)\langle X^2\rangle+\langle X^4\rangle+1=(a-1)^2+b^2+3, \end{align} where $\langle X^4\rangle=3$. So the best linear predictor is a constant $1$, which is on average how big $X^2$ is. Since $Y$ does not depend on $X$ linearly, any nonzero linear slope would make the predictor worse (see the $b^2$ term).

• Would $E[Y|X]=X^2$? So does that mean that $E[Y|X]=sin(x)$ in my example? – Silvia Rossi Oct 24 '17 at 3:16
• Yes. If the definition of "best" is minimum mean square error, the conditional expectation value is the best predictor. If the definition is changed to minimum mean absolute error, the conditional median would be the best predictor. In both cases the best predictor for $Y$ in my example is $X^2$ because the error distribution of $\epsilon$ is symmetric. Yeah, so is in your example. – Zhuoran He Oct 24 '17 at 3:18
• You can also consider $X\sim N(\mu,1)$ and $\,\epsilon\sim N(0,1)$. Then the best linear predictor would have a nonzero slope. But still, it is not as good as the best predictor. – Zhuoran He Oct 24 '17 at 3:28