# W and b for LMMSE using covariance(XY)

I would like to calculate the $W_{LMMSE}$ and $b_{LMMSE}$ for X which is a uniform random variable between $-\pi/2$ and $\pi/2$ and $Y=\sin(X)$. I have the following info:

$\Sigma_{XY} = 2/\pi$

$\Sigma_X = \pi^2/12$

$\mu_X=0$

I want to confirm if the following formula are correct?

$$W_{OLS} = \Sigma_{XY}/\Sigma_{X} = 24/\pi^3$$

and

$$b_{OLS} = \Sigma_{XY} - W_{OLS} \cdot \mu_X = 2/\pi$$

We have

$$(W_{LMMSE}, b_{LMMSE}) = \text{argmin}_{W, b} E[(Y-WX-b)^2]$$

I assume in this case Ordinary Least Square is same as Linear MMSE (Minimum Mean Squared Error)(LMMSE) (please correct me if I am wrong).

• additional link and resource where these formula for W and b could be find is really appreciated – Mona Jalal Apr 17 '18 at 1:19

Your second formula should be $$b_{OLS} = \mu_Y - W_{OLS}\cdot \mu_X = 0.$$ Indeed, let us find $(W_{LMMSE}, b_{LMMSE}) = \text{argmin}_{W, b} \mathop{\mathbb E}[(Y-WX-b)^2]$ by blunt frontal calculation. $$\mathop{\mathbb E}[(Y-WX-b)^2]=\mathop{\mathbb E}[Y^2]+W^2\mathop{\mathbb E}[X^2]+b^2-2W\mathop{\mathbb E}[XY]+2bW\mathop{\mathbb E}[X]-2b\mathop{\mathbb E}[Y].$$ Find the partial derivatives w.r.t $W$ and $b$ and equate them to zero: $$2W\mathop{\mathbb E}[X^2]-2\mathop{\mathbb E}[XY]+2b\mathop{\mathbb E}[X]=0$$ $$2b+2W\mathop{\mathbb E}[X]-2\mathop{\mathbb E}[Y]=0$$ Find $b=\mu_Y-W\cdot\mu_X$ from the second equation and substitute it to the first to find $W$. We get finally $$W_{OLS} = \frac{\mathop{\mathbb E}[XY] - \mathop{\mathbb E}[X]\mathop{\mathbb E}[Y]}{\mathop{\mathbb E}[X^2]-(\mathop{\mathbb E}[X])^2}=\frac{\Sigma_{XY}}{\Sigma_X},\quad b_{OLS}=\mu_Y - W_{OLS}\cdot\mu_X.$$