# Conditional expectation of multivariate normal

Note: In trying to formalize my question, I think I found one answer to it. I have still posted the question, in part because I hope that someone else has a less algebraically intensive solution.

The question: I'm reading Lamperti's Probability, Second Edition. I'm trying to understand Example 3 of Section 4 of Chapter 1 (page 25 of my book), which has to do with conditional expectations applied to multivariate normal distributions. He makes a leap of logic that I don't follow.

Lamperti says that, given random variables $$X_0, X_1, ..., X_n$$ with positive and continuous joint density $$f(t_0, t_1, ..., t_n)$$, we can write $$E[X_0 | X_1, ..., X_n]$$ as the random variable $$g(X_1, ..., X_n)$$, with $$g(t_1,...t_n)$$ defined as

$$g(t_1, ..., t_n) := \frac{\int s f(s, t_1, ..., t_n) ds}{\int f(s, t_1, ..., t_n)ds}$$

So far, so good. He also says that $$X_0, ..., X_n$$ follow a multivariate normal distribution as long as they have a joint density of the form

$$f(t_0, ..., t_n) = K \exp(-\frac{1}{2} \sum^n_{i,j=0}d_{ij}t_it_j)$$

with $$K$$ a normalizing constant and $$[d_{ij}]$$ a symmetric, positive-definite matrix. Also fine. Then, however, he says that, from the two facts above, we can deduce that $$E[X_0 | X_1, ..., X_n] = - \sum^n_{k=1} \frac{d_{k0}}{d_{00}}X_k$$

but I do not know how. Do you?

• You are right. After completing the square you get normal distribution with mean $-\sum_{i=1}^n t_i \frac{d_{i0}}{d_{00}}$. – NCh Jan 8 at 6:14
• You do not need to do all these ugly calculations. There is a much slicker and simpler argument by considering the conditional expectation as a projection in $L^2$. If you're interested I can post the answer. – badatmath Jan 8 at 9:37
• @badatmath Yes! Show me the slick argument. – Measure Theory Penguin Jan 8 at 17:01
• Just a few comments: (1) the above assumes at a minimum that $X_0$ has zero mean. In general I assume they are all zero mean. (2) The labelling of r.v.'s is arbitrary so I relabel them such that we want $E[X_{n+1}|(X_1,...,X_n)]$ (3) If you look at the covariance matrix, i.e. $D^{-1}$, and do Cholesky decomposition, then $\mathbf x = L\mathbf y$ where each $y_i$ is iid standard normal. This system is triangular with $y_i$'s as generators. We have an invertible linear map so $(X_1, ..., X_n)$ is qualitatively the same as $(Y_1, ..., Y_n)$. – user8675309 Jan 10 at 5:39
• Thus $E[X_{n+1}|(X_1,...,X_n)] =E[X_{n+1}|(Y_1,...,Y_n)] = \sum_{j=1}^{n+1}l_{n+1,j}E[ Y_j|(Y_1,...,Y_n)]$ $= \sum_{j=1}^{n}l_{n+1,j}E[ Y_j|(Y_1,...,Y_n)] = \sum_{j=1}^{n}l_{n,j+1} Y_j$ I.e. this implies the conditional expectation is a linear combination of the generators of $\{X_1, ..., X_n\}$. Converting between this formula and yours seems ugly but hopefully this gives some easy intuition for the relationship – user8675309 Jan 10 at 5:39

This is an algebraically intensive answer that depends on completing the square inside the exponential, which is a common trick when working with normal distributions. However, @badatmath has suggested a shorter, more enlightening approach.

Since $$f$$ appears beneath the integral sign in both the numerator and the denominator of $$g$$, any terms that can be both factored out of $$f$$ and pulled out from beneath the integral sign will cancel in the fraction. Therefore, any such terms can be ignored. To that end, let us rewrite $$f$$ in a way that factors out any terms not depending on $$t_0$$.

$$f(t_0, ... , t_n)$$ $$\propto \exp(-\frac{1}{2} \sum^n_{i,j=0}d_{ij}t_it_j)$$ $$= \exp(-\frac{1}{2} (\sum^n_{i=0 \lor j=0}d_{ij}t_it_j + \sum^n_{i \neq 0 \land j \neq 0}d_{ij}t_it_j))$$ $$\propto \exp(-\frac{1}{2} (\sum^n_{i=0 \lor j=0}d_{ij}t_it_j)$$ $$= \exp(-\sum^n_{i=1} d_{i0}t_it_0 -\frac{1}{2}d_{00}t_0^2)$$

Now, for reasons that will soon be apparent, let us complete the square. Specifically, let $$a^2 = \frac{1}{2} d_{00} t_0^2$$ and $$2ab = t_0 \sum^n_{i=1} d_{i0} t_i$$. Then $$a = \sqrt{\frac{1}{2} d_{00} t_0^2} = t_0 \sqrt{\frac{d_{00}}{2}}$$, (where $$d_{00} > 0$$ because $$[d_{ij}]$$ is positive definite), and

$$b$$ $$= \frac{2ab}{2a}$$ $$= \frac{t_0 \sum^n_{i=1} d_{i0} t_i}{t_0 \sqrt{2 d_{00}}}$$ $$= \frac{\sum^n_{i=1} d_{i0} t_i}{\sqrt{2 d_{00}}}$$

where $$b$$ does not depend on $$t_0$$. Thus,

$$f(t_0,...t_n)$$ $$\propto \exp(-\sum^n_{i=1} d_{i0}t_it_0 -\frac{1}{2}d_{00}t_0^2)$$ $$=\exp(-2ab - a^2)$$ $$=\exp(b^2 - b^2 -2ab - a^2)$$ $$=\exp(b^2 - (a+b)^2)$$ $$\propto \exp((a+b)^2)$$ $$= \exp((\sqrt{\frac{d_{00}}{2}}t_0 + \frac{\sum^n_{i=1} d_{i0} t_i}{\sqrt{2 d_{00}}})^2)$$ $$= \exp(\frac{d_{00}}{2}(t_0 + \frac{\sum^n_{i=1} d_{i0} t_i}{ d_{00}})^2)$$

which is just the kernel of a univariate normal density with variable $$t_0$$ and mean $$-\frac{\sum^n_{i=1} d_{i0} t_i}{ d_{00}}$$. Plugging the kernel into our equation for $$g$$, we find

$$g(t_1,...t_n)$$ $$=\frac{\int t_0 * \mathrm{kernel} * dt_0}{\int \mathrm{kernel} * dt_0}$$ $$=\frac{-\frac{\sum^n_{i=1} d_{i0} t_i}{ d_{00}}}{1}$$ $$=-\sum^n_{i=1} \frac{d_{i0}}{d_{00}}t_i$$

which is what we wanted to show.

• Might be simpler to write this as a change of variables. The exponent is $$-\frac {a_{11}} 2 x_1^2 - x_1 \sum_{i > 1} a_{1 i} x_i - S,$$ where $S$ does not depend on $x_1$. Then taking $x_1 = \tau + \alpha$ with $$\alpha = -\frac 1 {a_{11}} \sum_{i > 1} a_{1 i} x_i$$ gives $$\frac {\int_{\mathbb R} (\tau + \alpha) e^{C_1 \tau^2 + C_2} d\tau} {\int_{\mathbb R} e^{C_1 \tau^2 + C_2} d\tau} = \alpha.$$ – Maxim Jan 10 at 15:05