# Conditional expectation of gaussian variable

Suppose $$X = QZ + m + V$$ where $$Z \sim \mathcal{N}(0, I_K)$$, $$Q \in \mathbb{R}^{M\times K}$$, $$m \in \mathbb{R}^M$$ and $$V \sim \mathcal{N}(0, \sigma^2 I_M)$$. $$Z$$ and $$V$$ are independent.

I'm having trouble with one of the steps in a proof for the following expression for the expected value of $$Z$$ given $$X$$. $$E[Z\mid X=x] = (Q^TQ + \sigma^2 I_K)^{-1}Q^T(x-m).$$

The step I don't understand is: $$E[Z\mid X=x] = E[Z(X-m)^T]E[(X-m)(X-m)^T]^{-1}(x-m).$$

Is there a Bayes rule for expected values I can use? Can someone explain this step to me?

Edit: This is in the context of probabilistic principle component analysis. The data points $$x$$ are modeled as if they originated from a linearly transformed lower dimensional random variable ($$Z$$) plus high dimensional gaussian noise ($$V$$). So $$M \geq K$$ and $$Q$$ is assumed to be full rank.

• Isn't $M \geq K \wedge rg(Q) = K$ given as well? Feb 7, 2020 at 14:51
• $E[(X-m)(X-m)]$ doesn't make sense, since $X-m \in \mathbb{R^M}$ Feb 7, 2020 at 14:57
• $(QQ^T + \sigma I_K)^{-1}$ doesnt make sense as well. $\in \mathbb{R^{M \times K}} \Rightarrow QQ^T \in \mathbb{R^{M \times M}}$ and $\sigma I_K \in \mathbb{R^{K \times K}}$. Feb 7, 2020 at 16:06
• So is $M \leq K$ or $M \geq K$ correct? Feb 7, 2020 at 16:08
• $(Q^TQ + \sigma^2 I_M)^{-1} \in \mathbb{R^{M \times M}}$ and $Q^T \in \mathbb{R^{K \times M}}$ dont multiply as well. Feb 7, 2020 at 16:21

Suppose $$\left[ \begin{array}{c} T \\ U \end{array} \right] \sim \mathcal N\left( \left[ \begin{array}{c} 0 \\ 0 \end{array} \right], \left[ \begin{array}{cc} A & B \\ B^T & C \end{array} \right] \right) \tag 1$$ where $$A\in \mathbb R^{p\times p},$$ $$B\in\mathbb R^{p\times q},$$ $$C\in\mathbb R^{q\times q}$$ and the big matrix in $$(1)$$ is positive-definite. Then \begin{align} T\mid U \sim\mathcal N\left( BC^{-1} U , A - BC^{-1}B^T \right). \end{align}
Apply this in the case where $$T=Z,$$ $$U=X-m,$$ $$A=I_K,$$
\begin{align} \require{cancel} B = {} & \operatorname{cov}(Z, QZ+V) \\[10pt] = {} & \operatorname{cov}(Z, QZ) + {} \cancelto0{\operatorname{cov}(Z,V)\,\,\,} \\[10pt] = {} & \operatorname{cov}(Z,Z)Q^T \\ & \text{Here the Q gets transposed and gets} \\ & \text{pulled out on the right, not the left.} \\[10pt] = {} & Q^T, \\[15pt] \text{and } C = {} & QQ^T + \sigma^2 I_M. \end{align}
• Is the fact that $T|U \sim \mathcal{N}(BC^{-1}U, A - BC^{-1}B')$ a well known fact? Is there no intuitive explanation for why $E[Z\mid X=x] = E[Z(X-m)^T]E[(X-m)(X-m)^T]^{-1}(x-m)$? Feb 7, 2020 at 19:57
• @MauritsDescamps : It's a standard result about the multivariate normal distribution. In the case where $(T,U)$ is bivariate normal, it reduces to $$\operatorname E\left( \frac{T - \operatorname E T}{\operatorname{s.d.}(T)} \mid U \right) = \rho \cdot \frac{U - \operatorname E U}{\operatorname{s.d.} (U)},$$ i.e. multiply the z-score for $U$ by the correlation $\rho$ to get the expected z-score for $T.$ And the proportion of the variance of $T$ that is "explained" by $U$ is $\rho^2,$ so $$\operatorname{var}(T\mid U) = (1-\rho^2)\operatorname{var}(T).$$ Feb 7, 2020 at 20:02