I would like to calculate the $W_{LMMSE}$ and $b_{LMMSE}$ for X which is a uniform random variable between $-\pi/2$ and $\pi/2$ and $Y=\sin(X)$. I have the following info:

$\Sigma_{XY} = 2/\pi$

$\Sigma_X = \pi^2/12$


I want to confirm if the following formula are correct?

$$W_{OLS} = \Sigma_{XY}/\Sigma_{X} = 24/\pi^3$$


$$b_{OLS} = \Sigma_{XY} - W_{OLS} \cdot \mu_X = 2/\pi$$

We have

$$(W_{LMMSE}, b_{LMMSE}) = \text{argmin}_{W, b} E[(Y-WX-b)^2] $$

I assume in this case Ordinary Least Square is same as Linear MMSE (Minimum Mean Squared Error)(LMMSE) (please correct me if I am wrong).

  • $\begingroup$ additional link and resource where these formula for W and b could be find is really appreciated $\endgroup$ – Mona Jalal Apr 17 '18 at 1:19

Your second formula should be $$ b_{OLS} = \mu_Y - W_{OLS}\cdot \mu_X = 0. $$ Indeed, let us find $(W_{LMMSE}, b_{LMMSE}) = \text{argmin}_{W, b} \mathop{\mathbb E}[(Y-WX-b)^2]$ by blunt frontal calculation. $$ \mathop{\mathbb E}[(Y-WX-b)^2]=\mathop{\mathbb E}[Y^2]+W^2\mathop{\mathbb E}[X^2]+b^2-2W\mathop{\mathbb E}[XY]+2bW\mathop{\mathbb E}[X]-2b\mathop{\mathbb E}[Y]. $$ Find the partial derivatives w.r.t $W$ and $b$ and equate them to zero: $$ 2W\mathop{\mathbb E}[X^2]-2\mathop{\mathbb E}[XY]+2b\mathop{\mathbb E}[X]=0 $$ $$ 2b+2W\mathop{\mathbb E}[X]-2\mathop{\mathbb E}[Y]=0 $$ Find $b=\mu_Y-W\cdot\mu_X$ from the second equation and substitute it to the first to find $W$. We get finally $$ W_{OLS} = \frac{\mathop{\mathbb E}[XY] - \mathop{\mathbb E}[X]\mathop{\mathbb E}[Y]}{\mathop{\mathbb E}[X^2]-(\mathop{\mathbb E}[X])^2}=\frac{\Sigma_{XY}}{\Sigma_X},\quad b_{OLS}=\mu_Y - W_{OLS}\cdot\mu_X. $$

You can find these forumula in Wiki: https://en.wikipedia.org/wiki/Minimum_mean_square_error#Linear_MMSE_estimator


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.