# Linear MMSE estimate of MMSE estimator

This question is prompted by a recent discussion about the relationship between conditional expectation and covariance.

Suppose that $X$ and $Y$ are zero-mean unit-variance random variables with covariance (and correlation coefficient) $\rho$. The minimum-mean-square error (MMSE) estimator of $Y$ given $X$ is the random variable $g(X)$ that minimizes $E[(Y-g(X))^2]$, and as is well known, $$g(X) = E[Y \mid X] ~\text{minimizes}~E[(Y-g(X))^2]$$ It is also well known that $E[g(X)] = E[E[Y\mid X]] = E[Y] = 0$. In general, $g(X)$ is a nonlinear function. On the other hand, if the estimator is restricted to being of the form $\hat{Y} = aX + b$ where $a$ and $b$ are real numbers, then the linear MMSE estimator of $Y$ given $X$ is $\hat{Y} = \rho X$, that is, $$a = \rho, ~ b = 0, ~\text{minimizes}~E[(Y-aX-b)^2].$$ The linear MMSE estimator $\rho X$ has a mean-square-error $E[(Y-\rho X)^2] = 1 - \rho^2$ and so the mean-square-error of the MMSE estimator $g(X)$ can be no larger:
$$E[(Y-g(X))^2] \leq 1 - \rho^2.$$

A simplified version of the question in the previous discussion is: if $g(\cdot)$ is a decreasing function of its argument, show that $\rho$ is nonpositive.

My question is: what is the linear MMSE estimate of $g(X) = E[Y \mid X]$ given $X$? That is, what choice of real numbers $c$ and $d$ minimizes $E[(g(X) - cX - d)^2]$? Since $g(X)$ and $X$ both have zero mean and $X$ has unit variance, standard linear MMSE estimator theory gives that $d = 0$ and $$c = \frac{\text{cov}(g(X),X)}{\text{var}(X)} = \text{cov}(g(X),X) = E[Xg(X)]$$ which I think might work out to be $\rho$, but I am not sure about this. Any suggestions on how to proceed further would be appreciated.

-
Your final result is correct: for simple least-squares regression of $Y$ on $X$, the line passes through the point $(\mu_X,\mu_Y)$ with gradient $\rho \dfrac{\sigma_Y}{\sigma_X} = \dfrac{\text{cov}(X,Y)}{\sigma_X^2}$. So here it is just $\rho$. – Henry Dec 3 '11 at 3:36
I am somewhat uncomfortable with your language, since I fear that this way of using the word "linear" might feed into the popular misunderstanding that the reason why linear regression in called linear regression is that one is fitting a line. People who think that then find it confusing when a statistician insists that one is doing linear regression when one fits a parabola or a sine wave, etc. – Michael Hardy Dec 3 '11 at 5:38
@MichaelHardy I thought about editing the question to say something like "straight-line MMSE estimation" instead of "linear MMSE estimation" but decided against it because linear MMSE estimation is reasonably well-established, at least in the engineering literature: Google provides over $900,000$ hits. But, thanks for your answer which I am accepting. I was able to show $E[Xg(X)] = \rho$ for discrete and for jointly continuous random variables but wanted a proof that did not rely on special cases, and your answer gave me exactly what I wanted. – Dilip Sarwate Dec 4 '11 at 3:56

Your conjecture is correct. By the law of total expectation we have \begin{align} E(X(Y-g(X)) & = E(\;E(X(Y-g(X))\mid X)\;) \\ \\ & = E(\; E(XY\mid X) - E(Xg(X)\mid X)\;) \\ \\ & = E(\; XE(Y\mid X) - Xg(X) \;) \\ \\ & = E( Xg(X) - Xg(X)) = 0. \end{align} Therefore $$E(XY) = E(Xg(X)).$$