4
$\begingroup$

$X=Y+Z$ where $Y$ and $Z$ are independent Gaussians with zero means and variances $\sigma_Y^2$ and $\sigma_Z^2$ are known. How can I derive the MMSE and ML decision rules for estimating $Y$ for a given test point $X=x$ as a function of $x,\sigma_Y^2,\sigma_Z^2$?


I'm honestly lost on how to even begin. I know some probability theory, but don't have any experience doing proofs. If someone would point me in the right direction I'd really appreciate it.

$\endgroup$
2
  • 1
    $\begingroup$ Do you know how to find the conditional expected value of $Y$ given the event $X=x\text{ ?} \qquad$ $\endgroup$ Feb 2, 2017 at 5:30
  • $\begingroup$ I know the formula is the integral of y∗pmf(x|y)dy $\endgroup$
    – Austin
    Feb 4, 2017 at 2:43

1 Answer 1

3
+50
$\begingroup$

$\newcommand{\E}{\operatorname{E}}\newcommand{\var}{\operatorname{var}}$If a number $c$ is used as a prediction of the value of a yet unobserved random variable $W$, then the mean squared prediction error $\operatorname{E}((W-c)^2)$ is smaller if $c=\operatorname{E}(W)$ than if $c$ is any other number. This can be seen as follows: Let $\mu= \operatorname{E}(W);$ then \begin{align} \operatorname{E}((W-c)^2) & = \operatorname{E}(((W-\mu) + (\mu-c))^2) \\[10pt] & = \operatorname{E}((W-\mu)^2 + 2(W-\mu)(\mu-c) + (\mu-c)^2) \\[10pt] & = \operatorname{E}((W-\mu)^2) + \operatorname{E}(2(W-\mu)(\mu-c)) + \operatorname{E}((\mu-c)^2) \\[10pt] & = \operatorname{var}(W) + 2(\mu-c)\operatorname{E}(W-\mu) + (\mu-c)^2 \\ & \qquad\qquad\qquad \text{since $2(\mu-c)$ and $(\mu-c)^2$ are constant, i.e. not random} \\[12pt] & = \operatorname{var}(W) + 0 + (\mu-c)^2. \end{align} This depends on $c$ only through the last term, and the last term is $0$ if $c=\mu$ and is positive if $c={}$anything else.

Therefore what you need is $\operatorname{E}(Y\mid X=x),$ the conditional expected value of $Y$ given the event that $X=x.$ There are several ways one could approach the question of finding $\operatorname{E}(Y\mid X=x).$

One way involves the joint density of $Y$ and $Z$: $$ f_{Y,X}(y,z) = \text{constant} \cdot e^{-(y^2+z^2)/2}. $$ We are conditioning on the sum $x=y+z.$ Look at the straight line in the $(y,z)$-plane where their sum is $x$. Its slope is $-1$. The level sets of the joint density are circles $y^2+z^2=\text{radius}^2$ and the density gets larger as the circle gets smaller. The line $y+z=\text{something}$ touches one of those circles at a single point and crosses each of the larger circles at two points. The circle that it touches at just one point is on the line $z=y$, which meets the line $y+z=\text{something}$ at a right angle. That one point where it touches is $(y,z)=(x/2,x/2).$ That is the point where the joint density reaches its largest value on the line $y+z=\text{something}.$ By symmetry of the joint density, that is the point on that line representing the average value of $(Y,Z)$ given that $(Y,Z)$ is on that line. In other words, $\operatorname{E}((Y,Z)\mid Y+Z=x) = (x/2,x/2).$ So $\operatorname{E}(Y\mid X=x) = x/2.$

Another approach involves guessing based on the well behaved nature of the Gaussian distribution that $\operatorname{E}(Y\mid X=x)$ is some straight-line function of $x$ and then figuring out which line and then finding an argument to confirm the guess. A simple intuition should suggest $\operatorname{E}(Y\mid X=0) = 0,$ so let us guess that $\operatorname{E}(Y\mid X=x) = mx,$ and then try to figure out what number $m$ should be. One should have $\operatorname{E}(Y\mid X) = mX,$ so $\operatorname{E}(Y-mX\mid X) = 0.$

If the value of $X$ is observed, then $mX$ is the predicted value of $Y$, so $Y-mX$ is the residual -- the amount by which the actual value differs from the predicted value. The residual will be independent of the prediction only if the covariance between the residual and the prediction is $0$. We have $$ \operatorname{cov}(Y-mX, X) = \operatorname{cov}(Y-m(Y+Z),Y+Z) = 1-2m $$ and this is $0$ precisely when $m=\dfrac 1 2.$

If we can show that $Y-\dfrac 1 2 X$ is independent of $X$, then we have $\operatorname{E}\left( Y - \dfrac 1 2 X \mid X\right) = Y - \dfrac 1 2 X,$ and also $\operatorname{E}\left(\dfrac 1 2 X \mid X\right) = \dfrac 1 2 X$, and consequently $$ \operatorname{E}(Y\mid X) = \frac 1 2 X. $$ The covariance between $Y-\dfrac 1 2 X$ and $X$ is $0$. When both variables are linear combinations of independent Gaussians (in this case $Y$ and $Z$), then that is enough to infer independence.

Perhaps this is the best explanation I can give without either further thought or going into the theory of multivariate normal distributions.

Here's a point of view using matrix algebra: For a random vector $U\in\mathbb R^{n\times 1}$ $$ \var(U) = \E\Big((U - \E U) (U - \E U)^T \Big) \in \mathbb R^{n\times n}. $$ Feller calls this the variance; many authors call it the covariance matrix or just the covariance because its entries are the covariances between the scalar components of $U$.

For $A\in \mathbb R^{k\times n}$ we have $AU$ a random vector in $\mathbb R^{k\times 1}$ and $$ \var(AU) = A \Big( \var(U) \Big) A^T \in\mathbb R^{k\times k}. \tag 1 $$ So $$ \var\begin{bmatrix} Y \\ Z \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} $$ \begin{align} \var\begin{bmatrix} Y \\ X \end{bmatrix} & = \var\left( \begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} Y \\ Z \end{bmatrix} \right) \\[10pt] & = \begin{bmatrix} 1 & 1 \\ 1 & 2 \end{bmatrix} \text{ by }(1). \end{align} \begin{align} \var\begin{bmatrix} X \\ Y-\frac 1 2 X \end{bmatrix} = \begin{bmatrix} 0 & 1 \\ 1 & -1/2 \end{bmatrix}\begin{bmatrix} Y \\ X \end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 0 & 1/2 \end{bmatrix} \text{ by } (1). \end{align}

[ to be continued ]

$\endgroup$
3
  • $\begingroup$ Thanks I appreciate it. Gonna go through this in a couple hours. I've taken linear algebra, is matrix algebra another name for that or something different? $\endgroup$
    – Austin
    Feb 4, 2017 at 18:23
  • $\begingroup$ Is the following familiar to you? $$ \begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix} e & f \\ g & h \end{bmatrix} = \begin{bmatrix} ae+bg & af+bh \\ ce+dg & cf + dh \end{bmatrix} $$ $\endgroup$ Feb 4, 2017 at 18:27
  • $\begingroup$ Yup. I learned most of the basics and eigendecomposition $\endgroup$
    – Austin
    Feb 4, 2017 at 18:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.