# Conditional Expectation of Gaussian Vectors

Consider the random vector $$\mathbf{X}=(X_1,X_2,\ldots,X_n)$$. Assume that $$\mathbf{X}$$ is Gaussian with mean-vector $$\mathbf{0}$$ and covariance matrix $$C$$, that is $$\mathbf{X}\sim \mathcal{N}(\mathbf{0},C)$$.

In the simple case of two random variables $$(X_1,X_2)$$, we know that:

$$\mathbf{E}[X_2 | X_1] = \frac{\mathbf{E}(X_2 X_1)}{\mathbf{E}(X_1^2)}X_1 \tag{1}$$

So, the conditional expectation is a simple multiple of $$X_1$$.

We posit, that, in the case $$n$$ random variables, there exists constants $$a_1$$, $$a_2$$, $$\ldots$$, $$a_{n-1}$$, not all zero, such that:

$$\mathbf{E}[X_n | X_1\ldots X_{n-1}] = a_1X_1 + \ldots +a_{n-1}X_{n-1} \tag{2}$$

Now, we are interested to find the coefficients $$a_1,\ldots,a_{n-1}$$.

The book I am self-studying, applies the decomposition theorem, at this step. That is, a Gaussian vector $$\mathbf{X}=(X_1,X_2,\ldots,X_n)$$ with mean $$\mathbf{0}$$ and covariance matrix $$C$$ can be decomposed into $$n$$ IID standard normal random variables $$\mathbf{Z}=(Z_1,\ldots,Z_n)$$, $$Z_j \sim N(0,1)$$, and we can write:

$$\mathbf{X} = A \mathbf{Z}\tag{3}$$

or equivalently,

$$\mathbf{Z} = A^{-1}\mathbf{X} \tag{4}$$

We can find $$\mathbf{Z}$$ by performing the Gram-Schmidt orthogonalization of the random variables $$(X_1,\ldots,X_n)$$ (or cholesky factorization of $$C$$, $$C=AA^T$$). That is:

\begin{align*} Z_1' &= X_1 \\ Z_1 &= \frac{Z_1'}{\mathbf{E}[Z_1'^2]}\\ Z_2' &= X_2 - \mathbf{E}[X_2 Z_1] Z_1 \\ Z_2 &= \frac{Z_2'}{\mathbf{E}[Z_2'^2]}\\ \vdots\\ Z_n' &= X_n - \mathbf{E}[X_nZ_1]Z_1 - \ldots - \mathbf{E}[X_nZ_{n-1}]Z_{n-1} \tag{5} \end{align*}

We can check that the random variables $$Z_i$$, so constructed are orthogonal(independent), $$\mathbf{E}[Z_i Z_j] = 0, i\neq j$$.

Thus, we may as well write:

$$\mathbf{E}[X_n | X_1\ldots X_{n-1}] = b_1Z_1 + \ldots +b_{n-1}Z_{n-1} \tag{6}$$

Question. The author, then, suggests that $$b_j = \mathbf{E}[X_n Z_j]$$, where $$1\leq j \leq n-1$$. I'd like to ask for some help; how do I derive these $$b_j$$'s?

I've tried numerous things, I thought about it for nearly a day. But, I can't seem to derive it.

There is a much easier way. Write $$X_{n+1} = E(X_{n+1}\mid \vec X)+\zeta$$ where $$\vec X = (X_1,\dots,X_n),\quad \zeta = X_{n+1}-E(X_{n+1}\mid \vec X) = X_{n+1}-\alpha^T \vec X$$ The orthogonality relationship $$\mathrm{Cov}(\zeta,\vec X) = 0$$ implies $$\mathrm{Cov}(X_{n+1},\vec X) = \mathrm{Cov}(\vec X,\alpha^T \vec X) \implies \alpha = \mathrm{Cov}(\vec X)^{-1}\mathrm{Cov}(X_{n+1},\vec X)$$ This gives a guess for the conditional expectation $$E(X_{n+1}\mid \vec X)$$. To check that it’s correct, note that we have decomposed $$X_{n+1}$$ as the sum of a $$\vec X$$ measurable function and $$\zeta$$ orthogonal to $$\vec X$$. Uniqueness of orthogonal decomposition in Hilbert spaces completes the proof.