3
$\begingroup$

Let $n\ge2$ be an integer, let $\Sigma$ be a positive semidefinite, symmetric $n\times n$ matrix of real numbers partitioned as $$\Sigma=\begin{pmatrix}\Sigma_{a,a}&\Sigma_{a,b}\\\Sigma_{b,a}&\Sigma_{b,b}\end{pmatrix},$$ where $\Sigma_{a,a}$ is $1\times1$ and $\Sigma_{b,b}$ is $(n-1)\times(n-1),$ assume $\Sigma_{b,b}$ is positive definite (i.e., invertible) and let $X=(X_1,\dots,X_n)$ be $N(0,\Sigma),$ normal with mean zero and covariance matrix $\Sigma.$ I wish to find $E(X_1\mid X_2,\dots,X_n).$ In addition, I am using the Radon-Nikodym-derivative definition of conditional expectation, so I would prefer not to compute conditional densities $f_{X_a\mid X_b}(x_a\mid x_b)=f_{X_a,X_b}(x_a,x_b)/f_{X_b}(x_b).$

From Conditional Expectation Multivariate Normal, I can guess that $E(X_1\mid X_2,\dots,X_n)=\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T.$ To prove this result, I tried reasoning as follows, similar to user357269's answer to "Conditional expectation of a joint normal distribution": If $X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T$ and $\sigma(X_2,\dots,X_n)$ are independent, then we have $$E(X_1\mid X_2,\dots,X_n)$$ $$=E(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T+\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T\mid X_2,\dots,X_n)$$ $$=E(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T)+\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T=\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T,$$ where the last equality follows from $EX_1=0$ and $E((X_2,\dots,X_n))=0.$

However, I am stuck on showing independence. For the case $n=2,$ we can compute the covariance $\text{Cov}(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}X_2,X_2)=0$ and appeal to a theorem. However, I am unsure what to do for larger $n,$ since $(X_2,\dots,X_n)$ is vector-valued rather than real-valued.

$\endgroup$
5
  • $\begingroup$ Related: math.stackexchange.com/questions/291613/…. Also see stats.stackexchange.com/q/30588/119261, which has what you are looking for. $\endgroup$ Mar 25, 2020 at 7:50
  • $\begingroup$ @StubbornAtom I have already seen the CrossValidated post, but Macro's answer there is unappealing, since it assumes without proof that the conditional distribution is normal ("there's also a theorem that says all conditional distributions of a multivariate normal distribution are normal"). However, you can't (to my knowledge) prove that without computing the conditional density, which is cheating. $\endgroup$ Mar 25, 2020 at 12:55
  • $\begingroup$ @xFioraMstr18 This answer stats.stackexchange.com/a/270934/145128 derives the conditional expectation (confirming your guess) and proves that the conditional distribution is normal without resorting to computing any densities. $\endgroup$
    – grand_chat
    Mar 27, 2020 at 6:40
  • $\begingroup$ @grand_chat Hey it's you! Could you explain your proof for the case $\Sigma_{b,b}$ is non-invertible? You claimed $\Sigma_{a,b}-\Sigma_{a,b}\Sigma_{b,b}^+\Sigma_{b,b}=0,$ where $\Sigma_{b,b}^+$ is the Moore-Penrose pseudoinverse, but I had trouble verifying this result. I had actually asked a question, Moore-Penrose pseudoinverse: product on left with another matrix, about your answer just yesterday. $\endgroup$ Mar 27, 2020 at 12:29
  • $\begingroup$ @xFioraMstr18 Yep, still around. Please take a look at my answer to your question. $\endgroup$
    – grand_chat
    Mar 27, 2020 at 15:50

2 Answers 2

2
$\begingroup$

Let $V_1:=X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T$; then $X':=(V_1,X_2,\dots,X_n)$ is also Gaussian and its covariance matrix has the form $$ \Sigma'=\begin{pmatrix}\Sigma'_{a,a}&0 \\ 0&\Sigma_{b,b}\end{pmatrix}. $$ Writing the density of this new Gaussian vector, we can see that we can factorize with respect to the first variable and the independence between $V_1$ and $(X_2,\dots,X_n)$ follows.

$\endgroup$
1
$\begingroup$

There is a $1\times n$ matrix $A$ such that

$$ A(X_1,\ldots,X_n)^\top = \big(X_1-\Sigma_{a,b}\Sigma_{b,b}^{-1}(X_2,\dots,X_n)^T, \, X_2, \, X_3, \, \ldots, \, X_n\big) $$ The covariance matrix of $A(X_1,\ldots,X_n)^\top$ is $A\Sigma A^\top.$ If you observe that all entries in the first row and first column of this matrix are $0$ except the variance of the first component in the random vector, then that implies something about the factorization of the joint density function.

Let us look at a useful definition and a useful lemma:

Definition: $$ \operatorname{cov}\left( \left[ \begin{array}{c} Y_1 \\ \vdots \\ Y_m \end{array} \right], \left[ \begin{array}{c} X_1 \\ \vdots \\ X_n \end{array} \right] \right) = \text{a certain } m\times n \text{ matrix}. $$ (Details are an exercise.)

Definition: $$ \operatorname{var}\left[ \begin{array}{c} Y_1 \\ \vdots\\ Y_m \\ X_1 \\ \vdots \\ X_n \end{array} \right] = \text{a certain } (m+n)\times(m+n) \text{ matrix}. $$ The former matrix is found within the latter.

Lemma:

If the former matrix is the $m\times n$ zero matrix, then one can deduce something about factoring the multivariate normal density, and hence about independence.

You wrote:

For the case $n=2,$ we can compute the covariance ... and appeal to a theorem.

But it is not only in the case $n=2$ that that works.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.