Note: In trying to formalize my question, I think I found one answer to it. I have still posted the question, in part because I hope that someone else has a less algebraically intensive solution.

The question: I'm reading Lamperti's Probability, Second Edition. I'm trying to understand Example 3 of Section 4 of Chapter 1 (page 25 of my book), which has to do with conditional expectations applied to multivariate normal distributions. He makes a leap of logic that I don't follow.

Lamperti says that, given random variables $X_0, X_1, ..., X_n$ with positive and continuous joint density $f(t_0, t_1, ..., t_n)$, we can write $E[X_0 | X_1, ..., X_n]$ as the random variable $g(X_1, ..., X_n)$, with $g(t_1,...t_n)$ defined as

$$g(t_1, ..., t_n) := \frac{\int s f(s, t_1, ..., t_n) ds}{\int f(s, t_1, ..., t_n)ds}$$

So far, so good. He also says that $X_0, ..., X_n$ follow a multivariate normal distribution as long as they have a joint density of the form

$$ f(t_0, ..., t_n) = K \exp(-\frac{1}{2} \sum^n_{i,j=0}d_{ij}t_it_j)$$

with $K$ a normalizing constant and $[d_{ij}]$ a symmetric, positive-definite matrix. Also fine. Then, however, he says that, from the two facts above, we can deduce that $$E[X_0 | X_1, ..., X_n] = - \sum^n_{k=1} \frac{d_{k0}}{d_{00}}X_k$$

but I do not know how. Do you?

  • 1
    $\begingroup$ You are right. After completing the square you get normal distribution with mean $-\sum_{i=1}^n t_i \frac{d_{i0}}{d_{00}}$. $\endgroup$ – NCh Jan 8 at 6:14
  • 2
    $\begingroup$ You do not need to do all these ugly calculations. There is a much slicker and simpler argument by considering the conditional expectation as a projection in $L^2$. If you're interested I can post the answer. $\endgroup$ – badatmath Jan 8 at 9:37
  • $\begingroup$ @badatmath Yes! Show me the slick argument. $\endgroup$ – Measure Theory Penguin Jan 8 at 17:01
  • $\begingroup$ Just a few comments: (1) the above assumes at a minimum that $X_0$ has zero mean. In general I assume they are all zero mean. (2) The labelling of r.v.'s is arbitrary so I relabel them such that we want $E[X_{n+1}|(X_1,...,X_n)]$ (3) If you look at the covariance matrix, i.e. $D^{-1}$, and do Cholesky decomposition, then $\mathbf x = L\mathbf y$ where each $y_i$ is iid standard normal. This system is triangular with $y_i$'s as generators. We have an invertible linear map so $(X_1, ..., X_n)$ is qualitatively the same as $(Y_1, ..., Y_n)$. $\endgroup$ – user8675309 Jan 10 at 5:39
  • $\begingroup$ Thus $E[X_{n+1}|(X_1,...,X_n)] =E[X_{n+1}|(Y_1,...,Y_n)] = \sum_{j=1}^{n+1}l_{n+1,j}E[ Y_j|(Y_1,...,Y_n)] $ $= \sum_{j=1}^{n}l_{n+1,j}E[ Y_j|(Y_1,...,Y_n)] = \sum_{j=1}^{n}l_{n,j+1} Y_j$ I.e. this implies the conditional expectation is a linear combination of the generators of $\{X_1, ..., X_n\}$. Converting between this formula and yours seems ugly but hopefully this gives some easy intuition for the relationship $\endgroup$ – user8675309 Jan 10 at 5:39

This is an algebraically intensive answer that depends on completing the square inside the exponential, which is a common trick when working with normal distributions. However, @badatmath has suggested a shorter, more enlightening approach.

Since $f$ appears beneath the integral sign in both the numerator and the denominator of $g$, any terms that can be both factored out of $f$ and pulled out from beneath the integral sign will cancel in the fraction. Therefore, any such terms can be ignored. To that end, let us rewrite $f$ in a way that factors out any terms not depending on $t_0$.

$$f(t_0, ... , t_n)$$ $$\propto \exp(-\frac{1}{2} \sum^n_{i,j=0}d_{ij}t_it_j)$$ $$= \exp(-\frac{1}{2} (\sum^n_{i=0 \lor j=0}d_{ij}t_it_j + \sum^n_{i \neq 0 \land j \neq 0}d_{ij}t_it_j))$$ $$\propto \exp(-\frac{1}{2} (\sum^n_{i=0 \lor j=0}d_{ij}t_it_j)$$ $$= \exp(-\sum^n_{i=1} d_{i0}t_it_0 -\frac{1}{2}d_{00}t_0^2)$$

Now, for reasons that will soon be apparent, let us complete the square. Specifically, let $a^2 = \frac{1}{2} d_{00} t_0^2$ and $2ab = t_0 \sum^n_{i=1} d_{i0} t_i$. Then $a = \sqrt{\frac{1}{2} d_{00} t_0^2} = t_0 \sqrt{\frac{d_{00}}{2}}$, (where $d_{00} > 0$ because $[d_{ij}]$ is positive definite), and

$$ b $$ $$ = \frac{2ab}{2a}$$ $$ = \frac{t_0 \sum^n_{i=1} d_{i0} t_i}{t_0 \sqrt{2 d_{00}}}$$ $$ = \frac{\sum^n_{i=1} d_{i0} t_i}{\sqrt{2 d_{00}}}$$

where $b$ does not depend on $t_0$. Thus,

$$f(t_0,...t_n)$$ $$\propto \exp(-\sum^n_{i=1} d_{i0}t_it_0 -\frac{1}{2}d_{00}t_0^2)$$ $$=\exp(-2ab - a^2)$$ $$=\exp(b^2 - b^2 -2ab - a^2)$$ $$=\exp(b^2 - (a+b)^2)$$ $$\propto \exp((a+b)^2)$$ $$= \exp((\sqrt{\frac{d_{00}}{2}}t_0 + \frac{\sum^n_{i=1} d_{i0} t_i}{\sqrt{2 d_{00}}})^2)$$ $$= \exp(\frac{d_{00}}{2}(t_0 + \frac{\sum^n_{i=1} d_{i0} t_i}{ d_{00}})^2)$$

which is just the kernel of a univariate normal density with variable $t_0$ and mean $-\frac{\sum^n_{i=1} d_{i0} t_i}{ d_{00}}$. Plugging the kernel into our equation for $g$, we find

$$g(t_1,...t_n)$$ $$=\frac{\int t_0 * \mathrm{kernel} * dt_0}{\int \mathrm{kernel} * dt_0}$$ $$=\frac{-\frac{\sum^n_{i=1} d_{i0} t_i}{ d_{00}}}{1}$$ $$=-\sum^n_{i=1} \frac{d_{i0}}{d_{00}}t_i$$

which is what we wanted to show.

  • $\begingroup$ Might be simpler to write this as a change of variables. The exponent is $$-\frac {a_{11}} 2 x_1^2 - x_1 \sum_{i > 1} a_{1 i} x_i - S,$$ where $S$ does not depend on $x_1$. Then taking $x_1 = \tau + \alpha$ with $$\alpha = -\frac 1 {a_{11}} \sum_{i > 1} a_{1 i} x_i$$ gives $$\frac {\int_{\mathbb R} (\tau + \alpha) e^{C_1 \tau^2 + C_2} d\tau} {\int_{\mathbb R} e^{C_1 \tau^2 + C_2} d\tau} = \alpha.$$ $\endgroup$ – Maxim Jan 10 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.