I need a bit of help understanding a step in the derivation of the expected value of $\bf{x x^{T}}$, that is, $E[\bf{x x^{T}}]$ with a Gaussian distribution.

By definition, using the D-dimensional Gaussian distribution, we have:

$$E[{\bf{x x^{T}}}]=\frac{1}{(2\pi)^{\frac{D}{2}}}\frac{1}{|\Sigma|^{\frac{1}{2}}} \int \exp{\left\{ -\frac{1}{2}(\bf{x-\mu)^T\Sigma^{-1}(x-}\mu) \right\}}{\bf{x x^{T}}}d\bf{x}$$

We substitute $\bf{x-\mu}$ by $\bf{z}$:

$$E[{\bf{x x^{T}}}]=\frac{1}{(2\pi)^{\frac{D}{2}}}\frac{1}{|\Sigma|^{\frac{1}{2}}} \int \exp{\left\{ -\frac{1}{2}{\bf{z}}^T\Sigma^{-1}{\bf{z}} \right\}}({\bf{z+\mu}})(\bf{z+\mu})^{T}d{\bf{z}}$$

From this equation, the part that is giving me trouble is the $\bf{zz^{T}}$ term:

$$\frac{1}{(2\pi)^{\frac{D}{2}}}\frac{1}{|\Sigma|^{\frac{1}{2}}} \int \exp{\left\{ -\frac{1}{2}{\bf{z}}^T\Sigma^{-1}{\bf{z}} \right\}}{\bf{z}}\bf{z}^{T}d{\bf{z}}$$

The book says:

Again, we can make use of the eigenvector expansion of the covariance matrix given by (2.45), together with the completeness of the set of eigenvectors, to write ${\bf{z}} = \sum_{j=1}^{D} y_{j}{\bf{u_{j}}}$, where $y_{j} = \bf{u_{j}^{T}z}$

The equation (2.45) is the eigenvector equation:

$${\bf{\Sigma u_{i}}}=\lambda_{i}{\bf{u_{i}}}$$

I know that we can express $\bf{z}$ using the eigenvectors of $\bf{\Sigma}$ but the summation above doesn't make much sense to me. By the way, $y_{j}=\bf{u_{j}^{T}z}$ comes from the exponent in the Gaussian distribution and works as a change of coordinates (from $x$ to $y$), so at least I understand that part.

Here is the relevant section in the book:

enter image description here

And this is the page referenced in the quote above:

enter image description here

So, can anyone explain that part to me?

Thanks in advance

  • $\begingroup$ How is the covariance $\Sigma$ defined in the first place? I am used to seeing (for any distribution) $\mu = E(x)$, $\Sigma = E ((x-\mu)(x-\mu)^T)$. $\endgroup$ – copper.hat Mar 5 '13 at 7:00
  • $\begingroup$ Sure. In the multidimensional case, $\Sigma$ is defined as $E[({\bf{x}}-E[{\bf{x}}])({\bf{x}}-E[{\bf{x}}])^{T}]$ $\endgroup$ – Robert Smith Mar 5 '13 at 7:07
  • $\begingroup$ So why not just do $E(xx^T) = E((x-mu)(x-mu)^T -\mu \mu^T + x \mu^T + \mu x^T) = \Sigma -\mu \mu^T +\mu \mu^T +\mu \mu^T = \Sigma + \mu \mu^T$. $\endgroup$ – copper.hat Mar 5 '13 at 7:10

I think your central question is how to write $z$ as $$z = \sum_{j=1}^D (u_j^t z) \, u_j = \sum_{j=1}^D \langle u_j, z \rangle u_j \quad (1)$$ This, you can see as follows. As you have a complete, orthonormal set of EVs $\{ u_j \}$ that span $\mathbb{R}^D$, you can write $$z = \sum_{j=1}^D \alpha_j u_j \quad (2)$$ for some $\alpha_i$. Applying $u_i$ to (2), you get $$\begin{align} \langle u_i, z \rangle &= \sum_{j=1}^D\alpha_j \langle u_i, u_j \rangle \quad \text{using (bi)linearity of } \langle .,.\rangle \\ & = \alpha_i \cdot 1 + \sum_{j \neq i} \alpha_j \cdot 0 \quad \text{using the orthonormality of the EVs}\\ &= \alpha_i, \end{align}$$ and if you plug this back into (2), you get (1).

Edit: At least I understood that you wanted that particular line explained. The exercises at the end would be mostly unnecessary to write down here. Except for the first (what he calls "Eigenvector equation"), they all are steps in showing the so-called "Eigendecomposition of a (symmetric) matrix (Spectral Theorem)" - symmetry coming in to allow for getting an orthonormal basis. You can write such matrices $$A = Q \Lambda Q^t,$$ with $Q$ being the matrix having as its columns the EVs of $A$, and $\Lambda$ being a diagonal matrix with the eigenvalues on its diagonal. When you look at it closely, this is (2.49) in your book. It's a standard result in linear algebra; proofs are online, eg here. This leaves only (2.45), which I've seen but if you really need it, I would have to think about first (and hope I'll remember/can re-prove).

  • $\begingroup$ Thanks! That's a very simple explanation but what about the quote "we can make use of the eigenvector expansion of the covariance matrix given by (2.45)"? Is that just misleading or is referring to other proof? $\endgroup$ – Robert Smith Mar 5 '13 at 7:32
  • $\begingroup$ @RobertSmith: It is probably referring to another proof, but see above: I'd have to first think about (2.45), and it would need proof first. What I use above is a handy standard trick whenever you're dealing with ON bases, so probably useful to remember for other purposes too. :) $\endgroup$ – gnometorule Mar 5 '13 at 7:36
  • 2
    $\begingroup$ @gnometorule +1. You might want to use \langle and \rangle. $\endgroup$ – Did Mar 5 '13 at 8:28
  • $\begingroup$ Well, your reasoning is extremely clear so it's enough for me. Thanks! $\endgroup$ – Robert Smith Mar 5 '13 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.