I am trying to prove the following:

Let $(X_1,\dots,X_n)$ be a Gaussian vector with mean 0 and covariance matrix B. Find the distribution of $E(X_1\mid X_2,\dots,X_n).$

I know in general for two Gaussian r.v. $X_1$ and $X_2$ we can show that $f_{X_1|X_2} (x_1|x_2) = \frac{1}{\sigma_{X_1} \sqrt{2 \pi (1-\rho^2)}}\exp\frac{(-(x_{1}- \rho(\sigma_{X_1}/ \sigma_{X_2})x_2)^2)}{2 \sigma^{2}_{X_1}(1- \rho^{2})}$ where $\rho$ is the correlation and

$E(X_{1}|X_{2})=\int x_{1} f_{X_1|X_2} (x_1|x_2)dx_{1}. $

How can I generalize this for a gaussian random vector of size $n$? Can I conclude on the distribution of $E(X_1|X_2,...X_n)$ based on the form of $f_{X_1|X_2,\dots,X_n} (x_1|x_2,\dots,x_n)$? For example, in the case of 2 Gaussian r.v we produce a normal r.v with variance $2 \sigma^{2}_{X_1}(1- \rho^{2})$ and mean $\rho(\sigma_{X_1}/ \sigma_{X_2})x_2$.


First, it's follows very quickly that the conditional distribution is normal.

$X = (X_1, ..., X_n) \sim N(0, B)$

So the joint pdf. is $f_X(x) = const_1. e^{-\frac{1}{2}x'B^{-1}x}$, where $const_1=\frac{1}{(2\pi)^{n/2}B^{1/2}}$.

We then have -

$$ \begin{array}{rcl}f_{X_1|X_2,...,X_n}(x_1|x_2, ..., x_n) &=&\frac{f_{X_1,X_2,...,X_n}(x_1,...,x_n)}{f_{X_2,...,X_n}(x_2,...,x_n)}\\ &=&\frac{const_1.e^{-\frac{1}{2}x'B^{-1}x}}{f_{X_2,...X_n}(x_2,...,x_n)}\\ &=&const_2.e^{-\frac{1}{2}x'B^{-1}x} \end{array}$$

Note 1: $const_2=\frac{const_1}{f_{X_2,...,X_n}(x_2,...,x_n)}$ is actually determined by $x_2,...,x_n$. But it is useful to think of it as a constant as far as the pdf of $x_1$ is concerned.

Note 2: Since this is a pdf, $const_2(x_2,...,x_n)$ is such that the total integral is 1. This is useful, and enables us to loose track of it as we can recover this anytime by just integrating this over $x_1$, since $const_2 = 1/\int_{x_1}e^{-\frac{1}{2}x'Bx}$.

Say $B^{-1}=D=\left(\begin{matrix}D_{11}&D_{12}&...&D_{1n}\\D_{21}&D_{22}& ...& D_{2n}\\&&...&\\ D_{n1}&D_{n2}&...&D_{nn}\end{matrix}\right)$.

Then note that $x'B^{-1}x=x_1^2D_{11}+2x_1 \sum_{r=2}^n D_{1r}x_r + \sum_{r=2}^n \sum_{s=2}^n x_r x_s D_{rs}$. This shows that the distribution of $X_1$ is indeed normal.

Moreover, we can also read off the distribution's mean and variance -

$$\begin{array}{rcl}x'B^{-1}x&=&x_1^2D_{11}+2x_1 \sum_{r=2}^n D_{1r}x_r + \sum_{r=2}^n \sum_{s=2}^n x_r x_s D_{rs}\\ &=&\frac{(x_1-\mu)^2}{\sigma^2} + c \end{array}$$

Where $\sigma = 1/\sqrt{D_{11}}$, $\mu=-\frac{\sum_{r=2}^n D_{1r}x_r}{\sqrt{D_{11}}}$.

The full distribution $f_{X_1|X_2,...,X_n}(x_1|x_2,...,x_n)$ is thus -

$$\begin{array}{rcl} f_{X_1|X_2,...,X_n}(x_1|x_2,...,x_n) &=&const_2.e^{-\frac{(x_1-\mu)^2}{2\sigma^2} + c}\\ &=&const_2.e^c.e^{-\frac{(x_1-\mu)^2}{2\sigma^2}}\\ \end{array}$$

So the conditional pdf is univariate normal $N\left(-\frac{\sum_{r=2}^n D_{1r}x_r}{\sqrt{D_{11}}}, \frac{1}{D_{11}}\right)$, where $D=B^{-1}$.

  • $\begingroup$ How do we get from the joint pdf to the conditional pdf? I guess I'm confused why the exponents are the same in the joint and conditional? With you final line of reasoning is the resulting random variable normal because we can show that exponent in the conditional pdf is the sum of normal random variables? $\endgroup$ – user75514 Mar 27 '17 at 19:57
  • $\begingroup$ Conditional pdf is $f(x_1|x_2) = f(x_1,x_2) / f(x_2)$ right? The $f(x_2)$ is absorbed in the $const_2$ as far as the distribution is concerned. Does it help? Let me edit my response to make it clearer. $\endgroup$ – KalEl Mar 27 '17 at 22:19
  • $\begingroup$ Thanks that would be very helpful! $\endgroup$ – user75514 Mar 27 '17 at 22:35
  • $\begingroup$ Done. Also derived the parameters (assuming I made no mistakes) of the conditional pdf as univariate normal. Hopefully the logic is clear? Let me know if any step seems off or not justified. $\endgroup$ – KalEl Mar 27 '17 at 22:42
  • $\begingroup$ @user75514 You are a member of MSE for more than three years. Why none of answers to your question is accepted by you? If the answer is satisfactory for you, you can check it "accepted" by pressing check mark left of the answer. $\endgroup$ – NCh Mar 28 '17 at 3:24

Partition the column vector $X:=(X_1, X_2,\ldots, X_n)^T$ into subvectors $X_a$ and $X_b$: $$ X = \left(\begin{matrix}X_a\\X_b\end{matrix}\right) $$ (in your case $X_a=X_1$ is univariate) and correspondingly partition the mean vector $\mu$ and covariance matrix $\Sigma$ of $X$: $$ \mu = \left(\begin{matrix}\mu_a\\ \mu_b\end{matrix}\right) $$ $$ \Sigma=\left(\begin{matrix}\Sigma_{a,a}&\Sigma_{a,b}\\\Sigma_{b,a}&\Sigma_{b,b}\end{matrix}\right)$$ Assume that $\Sigma_{b,b}$ is invertible. To find the conditional expectation $E(X_a\mid X_b)$, first find a matrix $C$ of constants such that $Z:=X_a- C X_b$ is uncorrelated with $X_b$. For this to be true we demand $$ 0= \operatorname{cov} (Z, X_b)=\operatorname{cov} (X_a - CX_b, X_b)=\Sigma_{a,b}-C\Sigma_{b,b}, $$ which yields $$ C=\Sigma_{a,b}\Sigma_{b,b}^{-1}. $$ Therefore $$ \begin{align} E(X_a\mid X_b)&=E(Z + C X_b\mid X_b)\\ &=E(Z\mid X_b) + CX_b\\ &\stackrel{(*)}=E(Z) + CX_b\\ &= E(X_a) + C(X_b - E(X_b))\\ &= \mu_a + \Sigma_{a,b}\Sigma_{b,b}^{-1}(X_b - \mu_b),\tag1 \end{align} $$ where in (*) we use the fact that $Z$ and $X_b$ are uncorrelated multivariate Gaussian random vectors and therefore are independent. From (1) we see that $E(X_a\mid X_b)$ is a linear combination of the elements of the Gaussian vector $X_b$, and therefore also is Gaussian.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.