# Conditional expectation for multivariate normal: Independence trick

Let $$n\ge2$$ be an integer, let $$\Sigma$$ be a positive semidefinite, symmetric $$n\times n$$ matrix of real numbers partitioned as $$\Sigma=\begin{pmatrix}\Sigma_{a,a}&\Sigma_{a,b}\\\Sigma_{b,a}&\Sigma_{b,b}\end{pmatrix},$$ where $$\Sigma_{a,a}$$ is $$1\times1$$ and $$\Sigma_{b,b}$$ is $$(n-1)\times(n-1),$$ assume $$\Sigma_{b,b}$$ is positive definite (i.e., invertible) and let $$X=(X_1,\dots,X_n)$$ be $$N(0,\Sigma),$$ normal with mean zero and covariance matrix $$\Sigma.$$ I wish to find $$E(X_1\mid X_2,\dots,X_n).$$ In addition, I am using the Radon-Nikodym-derivative definition of conditional expectation, so I would prefer not to compute conditional densities $$f_{X_a\mid X_b}(x_a\mid x_b)=f_{X_a,X_b}(x_a,x_b)/f_{X_b}(x_b).$$

From Conditional Expectation Multivariate Normal, I can guess that $$E(X_1\mid X_2,\dots,X_n)=\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T.$$ To prove this result, I tried reasoning as follows, similar to user357269's answer to "Conditional expectation of a joint normal distribution": If $$X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T$$ and $$\sigma(X_2,\dots,X_n)$$ are independent, then we have $$E(X_1\mid X_2,\dots,X_n)$$ $$=E(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T+\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T\mid X_2,\dots,X_n)$$ $$=E(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T)+\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T=\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T,$$ where the last equality follows from $$EX_1=0$$ and $$E((X_2,\dots,X_n))=0.$$

However, I am stuck on showing independence. For the case $$n=2,$$ we can compute the covariance $$\text{Cov}(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}X_2,X_2)=0$$ and appeal to a theorem. However, I am unsure what to do for larger $$n,$$ since $$(X_2,\dots,X_n)$$ is vector-valued rather than real-valued.

• Related: math.stackexchange.com/questions/291613/…. Also see stats.stackexchange.com/q/30588/119261, which has what you are looking for. Mar 25, 2020 at 7:50
• @StubbornAtom I have already seen the CrossValidated post, but Macro's answer there is unappealing, since it assumes without proof that the conditional distribution is normal ("there's also a theorem that says all conditional distributions of a multivariate normal distribution are normal"). However, you can't (to my knowledge) prove that without computing the conditional density, which is cheating. Mar 25, 2020 at 12:55
• @xFioraMstr18 This answer stats.stackexchange.com/a/270934/145128 derives the conditional expectation (confirming your guess) and proves that the conditional distribution is normal without resorting to computing any densities. Mar 27, 2020 at 6:40
• @grand_chat Hey it's you! Could you explain your proof for the case $\Sigma_{b,b}$ is non-invertible? You claimed $\Sigma_{a,b}-\Sigma_{a,b}\Sigma_{b,b}^+\Sigma_{b,b}=0,$ where $\Sigma_{b,b}^+$ is the Moore-Penrose pseudoinverse, but I had trouble verifying this result. I had actually asked a question, Moore-Penrose pseudoinverse: product on left with another matrix, about your answer just yesterday. Mar 27, 2020 at 12:29
• @xFioraMstr18 Yep, still around. Please take a look at my answer to your question. Mar 27, 2020 at 15:50

Let $$V_1:=X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T$$; then $$X':=(V_1,X_2,\dots,X_n)$$ is also Gaussian and its covariance matrix has the form $$\Sigma'=\begin{pmatrix}\Sigma'_{a,a}&0 \\ 0&\Sigma_{b,b}\end{pmatrix}.$$ Writing the density of this new Gaussian vector, we can see that we can factorize with respect to the first variable and the independence between $$V_1$$ and $$(X_2,\dots,X_n)$$ follows.

There is a $$1\times n$$ matrix $$A$$ such that

$$A(X_1,\ldots,X_n)^\top = \big(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T, \, X_2, \, X_3, \, \ldots, \, X_n\big)$$ The covariance matrix of $$A(X_1,\ldots,X_n)^\top$$ is $$A\Sigma A^\top.$$ If you observe that all entries in the first row and first column of this matrix are $$0$$ except the variance of the first component in the random vector, then that implies something about the factorization of the joint density function.

Let us look at a useful definition and a useful lemma:

Definition: $$\operatorname{cov}\left( \left[ \begin{array}{c} Y_1 \\ \vdots \\ Y_m \end{array} \right], \left[ \begin{array}{c} X_1 \\ \vdots \\ X_n \end{array} \right] \right) = \text{a certain } m\times n \text{ matrix}.$$ (Details are an exercise.)

Definition: $$\operatorname{var}\left[ \begin{array}{c} Y_1 \\ \vdots\\ Y_m \\ X_1 \\ \vdots \\ X_n \end{array} \right] = \text{a certain } (m+n)\times(m+n) \text{ matrix}.$$ The former matrix is found within the latter.

Lemma:

If the former matrix is the $$m\times n$$ zero matrix, then one can deduce something about factoring the multivariate normal density, and hence about independence.

You wrote:

For the case $$n=2,$$ we can compute the covariance ... and appeal to a theorem.

But it is not only in the case $$n=2$$ that that works.