3
$\begingroup$

Let $n\ge2$ be an integer, let $\Sigma$ be a positive semidefinite, symmetric $n\times n$ matrix of real numbers partitioned as $$\Sigma=\begin{pmatrix}\Sigma_{a,a}&\Sigma_{a,b}\\\Sigma_{b,a}&\Sigma_{b,b}\end{pmatrix},$$ where $\Sigma_{a,a}$ is $1\times1$ and $\Sigma_{b,b}$ is $(n-1)\times(n-1),$ assume $\Sigma_{b,b}$ is positive definite (i.e., invertible) and let $X=(X_1,\dots,X_n)$ be $N(0,\Sigma),$ normal with mean zero and covariance matrix $\Sigma.$ I wish to find $E(X_1\mid X_2,\dots,X_n).$ In addition, I am using the Radon-Nikodym-derivative definition of conditional expectation, so I would prefer not to compute conditional densities $f_{X_a\mid X_b}(x_a\mid x_b)=f_{X_a,X_b}(x_a,x_b)/f_{X_b}(x_b).$

From Conditional Expectation Multivariate Normal, I can guess that $E(X_1\mid X_2,\dots,X_n)=\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T.$ To prove this result, I tried reasoning as follows, similar to user357269's answer to "Conditional expectation of a joint normal distribution": If $X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T$ and $\sigma(X_2,\dots,X_n)$ are independent, then we have $$E(X_1\mid X_2,\dots,X_n)$$ $$=E(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T+\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T\mid X_2,\dots,X_n)$$ $$=E(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T)+\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T=\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T,$$ where the last equality follows from $EX_1=0$ and $E((X_2,\dots,X_n))=0.$

However, I am stuck on showing independence. For the case $n=2,$ we can compute the covariance $\text{Cov}(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}X_2,X_2)=0$ and appeal to a theorem. However, I am unsure what to do for larger $n,$ since $(X_2,\dots,X_n)$ is vector-valued rather than real-valued.

$\endgroup$
5
  • $\begingroup$ Related: math.stackexchange.com/questions/291613/…. Also see stats.stackexchange.com/q/30588/119261, which has what you are looking for. $\endgroup$ Mar 25, 2020 at 7:50
  • $\begingroup$ @StubbornAtom I have already seen the CrossValidated post, but Macro's answer there is unappealing, since it assumes without proof that the conditional distribution is normal ("there's also a theorem that says all conditional distributions of a multivariate normal distribution are normal"). However, you can't (to my knowledge) prove that without computing the conditional density, which is cheating. $\endgroup$ Mar 25, 2020 at 12:55
  • $\begingroup$ @xFioraMstr18 This answer stats.stackexchange.com/a/270934/145128 derives the conditional expectation (confirming your guess) and proves that the conditional distribution is normal without resorting to computing any densities. $\endgroup$
    – grand_chat
    Mar 27, 2020 at 6:40
  • $\begingroup$ @grand_chat Hey it's you! Could you explain your proof for the case $\Sigma_{b,b}$ is non-invertible? You claimed $\Sigma_{a,b}-\Sigma_{a,b}\Sigma_{b,b}^+\Sigma_{b,b}=0,$ where $\Sigma_{b,b}^+$ is the Moore-Penrose pseudoinverse, but I had trouble verifying this result. I had actually asked a question, Moore-Penrose pseudoinverse: product on left with another matrix, about your answer just yesterday. $\endgroup$ Mar 27, 2020 at 12:29
  • $\begingroup$ @xFioraMstr18 Yep, still around. Please take a look at my answer to your question. $\endgroup$
    – grand_chat
    Mar 27, 2020 at 15:50

2 Answers 2

2
$\begingroup$

Let $V_1:=X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T$; then $X':=(V_1,X_2,\dots,X_n)$ is also Gaussian and its covariance matrix has the form $$ \Sigma'=\begin{pmatrix}\Sigma'_{a,a}&0 \\ 0&\Sigma_{b,b}\end{pmatrix}. $$ Writing the density of this new Gaussian vector, we can see that we can factorize with respect to the first variable and the independence between $V_1$ and $(X_2,\dots,X_n)$ follows.

$\endgroup$
1
$\begingroup$

There is a $1\times n$ matrix $A$ such that

$$ A(X_1,\ldots,X_n)^\top = \big(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T, \, X_2, \, X_3, \, \ldots, \, X_n\big) $$ The covariance matrix of $A(X_1,\ldots,X_n)^\top$ is $A\Sigma A^\top.$ If you observe that all entries in the first row and first column of this matrix are $0$ except the variance of the first component in the random vector, then that implies something about the factorization of the joint density function.

Let us look at a useful definition and a useful lemma:

Definition: $$ \operatorname{cov}\left( \left[ \begin{array}{c} Y_1 \\ \vdots \\ Y_m \end{array} \right], \left[ \begin{array}{c} X_1 \\ \vdots \\ X_n \end{array} \right] \right) = \text{a certain } m\times n \text{ matrix}. $$ (Details are an exercise.)

Definition: $$ \operatorname{var}\left[ \begin{array}{c} Y_1 \\ \vdots\\ Y_m \\ X_1 \\ \vdots \\ X_n \end{array} \right] = \text{a certain } (m+n)\times(m+n) \text{ matrix}. $$ The former matrix is found within the latter.

Lemma:

If the former matrix is the $m\times n$ zero matrix, then one can deduce something about factoring the multivariate normal density, and hence about independence.

You wrote:

For the case $n=2,$ we can compute the covariance ... and appeal to a theorem.

But it is not only in the case $n=2$ that that works.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .