1
$\begingroup$

Suppose $$X = QZ + m + V$$ where $Z \sim \mathcal{N}(0, I_K)$, $Q \in \mathbb{R}^{M\times K}$, $m \in \mathbb{R}^M$ and $V \sim \mathcal{N}(0, \sigma^2 I_M)$. $Z$ and $V$ are independent.

I'm having trouble with one of the steps in a proof for the following expression for the expected value of $Z$ given $X$. $$E[Z\mid X=x] = (Q^TQ + \sigma^2 I_K)^{-1}Q^T(x-m).$$

The step I don't understand is: $$E[Z\mid X=x] = E[Z(X-m)^T]E[(X-m)(X-m)^T]^{-1}(x-m).$$

Is there a Bayes rule for expected values I can use? Can someone explain this step to me?

Edit: This is in the context of probabilistic principle component analysis. The data points $x$ are modeled as if they originated from a linearly transformed lower dimensional random variable ($Z$) plus high dimensional gaussian noise ($V$). So $M \geq K$ and $Q$ is assumed to be full rank.

$\endgroup$
11
  • 1
    $\begingroup$ Isn't $M \geq K \wedge rg(Q) = K$ given as well? $\endgroup$
    – Grada Gukovic
    Feb 7, 2020 at 14:51
  • 1
    $\begingroup$ $E[(X-m)(X-m)]$ doesn't make sense, since $X-m \in \mathbb{R^M}$ $\endgroup$
    – Grada Gukovic
    Feb 7, 2020 at 14:57
  • 1
    $\begingroup$ $(QQ^T + \sigma I_K)^{-1}$ doesnt make sense as well. $ \in \mathbb{R^{M \times K}} \Rightarrow QQ^T \in \mathbb{R^{M \times M}}$ and $ \sigma I_K \in \mathbb{R^{K \times K}}$. $\endgroup$
    – Grada Gukovic
    Feb 7, 2020 at 16:06
  • 1
    $\begingroup$ So is $M \leq K$ or $M \geq K$ correct? $\endgroup$
    – Grada Gukovic
    Feb 7, 2020 at 16:08
  • 1
    $\begingroup$ $(Q^TQ + \sigma^2 I_M)^{-1} \in \mathbb{R^{M \times M}}$ and $Q^T \in \mathbb{R^{K \times M}}$ dont multiply as well. $\endgroup$
    – Grada Gukovic
    Feb 7, 2020 at 16:21

1 Answer 1

Reset to default
1
$\begingroup$

Suppose $$ \left[ \begin{array}{c} T \\ U \end{array} \right] \sim \mathcal N\left( \left[ \begin{array}{c} 0 \\ 0 \end{array} \right], \left[ \begin{array}{cc} A & B \\ B^T & C \end{array} \right] \right) \tag 1 $$ where $A\in \mathbb R^{p\times p},$ $B\in\mathbb R^{p\times q},$ $C\in\mathbb R^{q\times q}$ and the big matrix in $(1)$ is positive-definite. Then \begin{align} T\mid U \sim\mathcal N\left( BC^{-1} U , A - BC^{-1}B^T \right). \end{align}

Apply this in the case where $T=Z,$ $U=X-m,$ $A=I_K,$

\begin{align} \require{cancel} B = {} & \operatorname{cov}(Z, QZ+V) \\[10pt] = {} & \operatorname{cov}(Z, QZ) + {} \cancelto0{\operatorname{cov}(Z,V)\,\,\,} \\[10pt] = {} & \operatorname{cov}(Z,Z)Q^T \\ & \text{Here the $Q$ gets transposed and gets} \\ & \text{pulled out on the right, not the left.} \\[10pt] = {} & Q^T, \\[15pt] \text{and } C = {} & QQ^T + \sigma^2 I_M. \end{align}

$\endgroup$
4
  • $\begingroup$ Is the fact that $T|U \sim \mathcal{N}(BC^{-1}U, A - BC^{-1}B')$ a well known fact? Is there no intuitive explanation for why $E[Z\mid X=x] = E[Z(X-m)^T]E[(X-m)(X-m)^T]^{-1}(x-m)$? $\endgroup$
    – MDescamps
    Feb 7, 2020 at 19:57
  • 1
    $\begingroup$ @MauritsDescamps : It's a standard result about the multivariate normal distribution. In the case where $(T,U)$ is bivariate normal, it reduces to $$ \operatorname E\left( \frac{T - \operatorname E T}{\operatorname{s.d.}(T)} \mid U \right) = \rho \cdot \frac{U - \operatorname E U}{\operatorname{s.d.} (U)}, $$ i.e. multiply the z-score for $U$ by the correlation $\rho$ to get the expected z-score for $T.$ And the proportion of the variance of $T$ that is "explained" by $U$ is $\rho^2,$ so $$ \operatorname{var}(T\mid U) = (1-\rho^2)\operatorname{var}(T). $$ $\endgroup$
    – Michael Hardy
    Feb 7, 2020 at 20:02
  • $\begingroup$ @MauritsDescamps : Possibly a posted question could ask how to prove this standard result. Or possibly it's already here. $\endgroup$
    – Michael Hardy
    Feb 7, 2020 at 20:21
  • $\begingroup$ I just realized that this is just the minimum mean square error estimate, with the covariance and variance. That's intuitive enough for me. $\endgroup$
    – MDescamps
    Feb 7, 2020 at 20:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .