Consider the random vector $\mathbf{X}=(X_1,X_2,\ldots,X_n)$. Assume that $\mathbf{X}$ is Gaussian with mean-vector $\mathbf{0}$ and covariance matrix $C$, that is $\mathbf{X}\sim \mathcal{N}(\mathbf{0},C)$.

In the simple case of two random variables $(X_1,X_2)$, we know that:

$$\mathbf{E}[X_2 | X_1] = \frac{\mathbf{E}(X_2 X_1)}{\mathbf{E}(X_1^2)}X_1 \tag{1}$$

So, the conditional expectation is a simple multiple of $X_1$.

We posit, that, in the case $n$ random variables, there exists constants $a_1$, $a_2$, $\ldots$, $a_{n-1}$, not all zero, such that:

$$\mathbf{E}[X_n | X_1\ldots X_{n-1}] = a_1X_1 + \ldots +a_{n-1}X_{n-1} \tag{2}$$

Now, we are interested to find the coefficients $a_1,\ldots,a_{n-1}$.

The book I am self-studying, applies the decomposition theorem, at this step. That is, a Gaussian vector $\mathbf{X}=(X_1,X_2,\ldots,X_n)$ with mean $\mathbf{0}$ and covariance matrix $C$ can be decomposed into $n$ IID standard normal random variables $\mathbf{Z}=(Z_1,\ldots,Z_n)$, $Z_j \sim N(0,1)$, and we can write:

$$\mathbf{X} = A \mathbf{Z}\tag{3}$$

or equivalently,

$$\mathbf{Z} = A^{-1}\mathbf{X} \tag{4}$$

We can find $\mathbf{Z}$ by performing the Gram-Schmidt orthogonalization of the random variables $(X_1,\ldots,X_n)$ (or cholesky factorization of $C$, $C=AA^T$). That is:

\begin{align*} Z_1' &= X_1 \\ Z_1 &= \frac{Z_1'}{\mathbf{E}[Z_1'^2]}\\ Z_2' &= X_2 - \mathbf{E}[X_2 Z_1] Z_1 \\ Z_2 &= \frac{Z_2'}{\mathbf{E}[Z_2'^2]}\\ \vdots\\ Z_n' &= X_n - \mathbf{E}[X_nZ_1]Z_1 - \ldots - \mathbf{E}[X_nZ_{n-1}]Z_{n-1} \tag{5} \end{align*}

We can check that the random variables $Z_i$, so constructed are orthogonal(independent), $\mathbf{E}[Z_i Z_j] = 0, i\neq j$.

Thus, we may as well write:

$$\mathbf{E}[X_n | X_1\ldots X_{n-1}] = b_1Z_1 + \ldots +b_{n-1}Z_{n-1} \tag{6}$$

Question. The author, then, suggests that $b_j = \mathbf{E}[X_n Z_j]$, where $1\leq j \leq n-1$. I'd like to ask for some help; how do I derive these $b_j$'s?

I've tried numerous things, I thought about it for nearly a day. But, I can't seem to derive it.


1 Answer 1


There is a much easier way. Write $$X_{n+1} = E(X_{n+1}\mid \vec X)+\zeta$$ where $$ \vec X = (X_1,\dots,X_n),\quad \zeta = X_{n+1}-E(X_{n+1}\mid \vec X) = X_{n+1}-\alpha^T \vec X$$ The orthogonality relationship $$\mathrm{Cov}(\zeta,\vec X) = 0$$ implies $$\mathrm{Cov}(X_{n+1},\vec X) = \mathrm{Cov}(\vec X,\alpha^T \vec X) \implies \alpha = \mathrm{Cov}(\vec X)^{-1}\mathrm{Cov}(X_{n+1},\vec X)$$ This gives a guess for the conditional expectation $E(X_{n+1}\mid \vec X)$. To check that it’s correct, note that we have decomposed $X_{n+1}$ as the sum of a $\vec X$ measurable function and $\zeta$ orthogonal to $\vec X$. Uniqueness of orthogonal decomposition in Hilbert spaces completes the proof.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .