6
$\begingroup$

I am trying to prove the following:

Let $(X_1,\dots,X_n)$ be a Gaussian vector with mean 0 and covariance matrix B. Find the distribution of $E(X_1\mid X_2,\dots,X_n).$

I know in general for two Gaussian r.v. $X_1$ and $X_2$ we can show that $f_{X_1|X_2} (x_1|x_2) = \frac{1}{\sigma_{X_1} \sqrt{2 \pi (1-\rho^2)}}\exp\frac{(-(x_{1}- \rho(\sigma_{X_1}/ \sigma_{X_2})x_2)^2)}{2 \sigma^{2}_{X_1}(1- \rho^{2})}$ where $\rho$ is the correlation and

$E(X_{1}|X_{2})=\int x_{1} f_{X_1|X_2} (x_1|x_2)dx_{1}. $

How can I generalize this for a gaussian random vector of size $n$? Can I conclude on the distribution of $E(X_1|X_2,...X_n)$ based on the form of $f_{X_1|X_2,\dots,X_n} (x_1|x_2,\dots,x_n)$? For example, in the case of 2 Gaussian r.v we produce a normal r.v with variance $2 \sigma^{2}_{X_1}(1- \rho^{2})$ and mean $\rho(\sigma_{X_1}/ \sigma_{X_2})x_2$.

$\endgroup$

2 Answers 2

9
$\begingroup$

Partition the column vector $X:=(X_1, X_2,\ldots, X_n)^T$ into subvectors $X_a$ and $X_b$: $$ X = \left(\begin{matrix}X_a\\X_b\end{matrix}\right) $$ (in your case $X_a=X_1$ is univariate) and correspondingly partition the mean vector $\mu$ and covariance matrix $\Sigma$ of $X$: $$ \mu = \left(\begin{matrix}\mu_a\\ \mu_b\end{matrix}\right) $$ $$ \Sigma=\left(\begin{matrix}\Sigma_{a,a}&\Sigma_{a,b}\\\Sigma_{b,a}&\Sigma_{b,b}\end{matrix}\right)$$ Assume that $\Sigma_{b,b}$ is invertible. To find the conditional expectation $E(X_a\mid X_b)$, first find a matrix $C$ of constants such that $Z:=X_a- C X_b$ is uncorrelated with $X_b$. For this to be true we demand $$ 0= \operatorname{cov} (Z, X_b)=\operatorname{cov} (X_a - CX_b, X_b)=\Sigma_{a,b}-C\Sigma_{b,b}, $$ which yields $$ C=\Sigma_{a,b}\Sigma_{b,b}^{-1}. $$ Therefore $$ \begin{align} E(X_a\mid X_b)&=E(Z + C X_b\mid X_b)\\ &=E(Z\mid X_b) + CX_b\\ &\stackrel{(*)}=E(Z) + CX_b\\ &= E(X_a) + C(X_b - E(X_b))\\ &= \mu_a + \Sigma_{a,b}\Sigma_{b,b}^{-1}(X_b - \mu_b),\tag1 \end{align} $$ where in (*) we use the fact that $Z$ and $X_b$ are uncorrelated multivariate Gaussian random vectors and therefore are independent. From (1) we see that $E(X_a\mid X_b)$ is a linear combination of the elements of the Gaussian vector $X_b$, and therefore also is Gaussian.

$\endgroup$
3
  • $\begingroup$ In the computation of $cov(X_{a} -CX_{b},X_{b})$ shouldn't it be $C^{2}$ instead of $C$? Since if I recall correctly, $cov(X) =E[XX^{T}]-E[X]E[X]^{T}$. $\endgroup$
    – user135520
    Mar 11, 2021 at 18:53
  • $\begingroup$ To calculate the cross-covariance $\operatorname{cov}(CX_b, X_b)$, use $\operatorname{cov}(X,Y)=E(XY^T) - E(X)E(Y^T)$ with $X:=CX_b$ and $Y:=X_b$. The $C$ factors out of the expectation, using $E(CW)=CE(W)$. So the result has just one $C$. $\endgroup$
    – grand_chat
    Mar 11, 2021 at 19:22
  • $\begingroup$ Oh right, my apologies, thank you, I thought $Y:=CX_{b}$ as well. $\endgroup$
    – user135520
    Mar 11, 2021 at 19:32
3
$\begingroup$

First, it's follows very quickly that the conditional distribution is normal.

$X = (X_1, ..., X_n) \sim N(0, B)$

So the joint pdf. is $f_X(x) = const_1. e^{-\frac{1}{2}x'B^{-1}x}$, where $const_1=\frac{1}{(2\pi)^{n/2}B^{1/2}}$.

We then have -

$$ \begin{array}{rcl}f_{X_1|X_2,...,X_n}(x_1|x_2, ..., x_n) &=&\frac{f_{X_1,X_2,...,X_n}(x_1,...,x_n)}{f_{X_2,...,X_n}(x_2,...,x_n)}\\ &=&\frac{const_1.e^{-\frac{1}{2}x'B^{-1}x}}{f_{X_2,...X_n}(x_2,...,x_n)}\\ &=&const_2.e^{-\frac{1}{2}x'B^{-1}x} \end{array}$$

Note 1: $const_2=\frac{const_1}{f_{X_2,...,X_n}(x_2,...,x_n)}$ is actually determined by $x_2,...,x_n$. But it is useful to think of it as a constant as far as the pdf of $x_1$ is concerned.

Note 2: Since this is a pdf, $const_2(x_2,...,x_n)$ is such that the total integral is 1. This is useful, and enables us to loose track of it as we can recover this anytime by just integrating this over $x_1$, since $const_2 = 1/\int_{x_1}e^{-\frac{1}{2}x'Bx}$.

Say $B^{-1}=D=\left(\begin{matrix}D_{11}&D_{12}&...&D_{1n}\\D_{21}&D_{22}& ...& D_{2n}\\&&...&\\ D_{n1}&D_{n2}&...&D_{nn}\end{matrix}\right)$.

Then note that $x'B^{-1}x=x_1^2D_{11}+2x_1 \sum_{r=2}^n D_{1r}x_r + \sum_{r=2}^n \sum_{s=2}^n x_r x_s D_{rs}$. This shows that the distribution of $X_1$ is indeed normal.

Moreover, we can also read off the distribution's mean and variance -

$$\begin{array}{rcl}x'B^{-1}x&=&x_1^2D_{11}+2x_1 \sum_{r=2}^n D_{1r}x_r + \sum_{r=2}^n \sum_{s=2}^n x_r x_s D_{rs}\\ &=&\frac{(x_1-\mu)^2}{\sigma^2} + c \end{array}$$

Where $\sigma = 1/\sqrt{D_{11}}$, $\mu=-\frac{\sum_{r=2}^n D_{1r}x_r}{\sqrt{D_{11}}}$.

The full distribution $f_{X_1|X_2,...,X_n}(x_1|x_2,...,x_n)$ is thus -

$$\begin{array}{rcl} f_{X_1|X_2,...,X_n}(x_1|x_2,...,x_n) &=&const_2.e^{-\frac{(x_1-\mu)^2}{2\sigma^2} + c}\\ &=&const_2.e^c.e^{-\frac{(x_1-\mu)^2}{2\sigma^2}}\\ \end{array}$$

So the conditional pdf is univariate normal $N\left(-\frac{\sum_{r=2}^n D_{1r}x_r}{\sqrt{D_{11}}}, \frac{1}{D_{11}}\right)$, where $D=B^{-1}$.

$\endgroup$
6
  • $\begingroup$ How do we get from the joint pdf to the conditional pdf? I guess I'm confused why the exponents are the same in the joint and conditional? With you final line of reasoning is the resulting random variable normal because we can show that exponent in the conditional pdf is the sum of normal random variables? $\endgroup$
    – user75514
    Mar 27, 2017 at 19:57
  • $\begingroup$ Conditional pdf is $f(x_1|x_2) = f(x_1,x_2) / f(x_2)$ right? The $f(x_2)$ is absorbed in the $const_2$ as far as the distribution is concerned. Does it help? Let me edit my response to make it clearer. $\endgroup$
    – KalEl
    Mar 27, 2017 at 22:19
  • $\begingroup$ Thanks that would be very helpful! $\endgroup$
    – user75514
    Mar 27, 2017 at 22:35
  • $\begingroup$ Done. Also derived the parameters (assuming I made no mistakes) of the conditional pdf as univariate normal. Hopefully the logic is clear? Let me know if any step seems off or not justified. $\endgroup$
    – KalEl
    Mar 27, 2017 at 22:42
  • $\begingroup$ @user75514 You are a member of MSE for more than three years. Why none of answers to your question is accepted by you? If the answer is satisfactory for you, you can check it "accepted" by pressing check mark left of the answer. $\endgroup$
    – NCh
    Mar 28, 2017 at 3:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .