Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm a bit stumped by the exponential family representation of a multi-variate Gaussian distribution. Basically, the exponential form is a generic form for a large class of probability distributions. The standard form is

$$f_X(x) = \exp[\theta' T(x) + F(\theta)]$$

where $\theta$ is a set of parameters (based on $\mu$ and $\Sigma$), $T(x)$ is a vector of sufficient statistics, and $F$ is a function of the parameters that ensures the distribution is a pdf, i.e., sums to one. For more information on this form, see http://www.cs.columbia.edu/~jebara/4771/tutorials/lecture12.pdf, http://en.wikipedia.org/wiki/Exponential_family, etc.

The "conversion" for a multi-variate Gaussian distribution to exponential family form is listed as

$$\theta = [\Sigma^{-1}\mu, -\frac{1}{2}\Sigma^{-1}]'$$ $$T(x) = [x, x x']'$$

but this is confusing because the outer product $x x'$ is a matrix and $-\frac{1}{2}\Sigma^{-1}$ is also a matrix. Thus, it seems the product between $\theta$ and $T(x)$ should result in a scalar "entry" and a matrix "entry". Obviously, this expression needs to evaluate to a scalar.

The inner product works fine in the scalar case, and I understand this conversion is computed by manipulating to the quadratic form $x'Ax + b'x$. Still, it seems that I am completely missing something here. Thanks for your help.

share|cite|improve this question
1  
Consider the "vectorized" version of $xx'$, then you'll see it matches the form. Or, just write it out in summation notation. Let $S = \Sigma^{-1}$ and $s_{ij}$ be the $(i,j)$th element of $S$. Define $y_{ij} = x_i x_j$. Let $u_i$ be the $i$th element of $S \mu$. Then the exponent of the density is $-\frac{1}{2} \sum_i \sum_j s_{ij} y_{ij} + \sum_i u_i x_i + h(S,u)$ which has the required form. You need to know that the the $y_{ij}$ and $x_i$ are linearly independent as well as the $s_{ij}$ and $u_i$ (so that you know you're not dealing with a curved exponential family. That's not hard. – cardinal Mar 13 '11 at 3:38
    
@cardinal Indeed, That's not hard, but @RandomGuy is right to point out there is at least an abuse of notation here, since what is written is a matrix product and what is meant is the scalar product of two $n\times n$ matrices transformed into vectors of size $n^2$ (as you aptly explain). If only every question on MSE could be as relevant as this one! – Did Mar 13 '11 at 10:12
    
@Didier, @RandomGuy, sorry I ran out of characters in my comment. My last sentence was supposed to say: "This last part is not hard to show." I truncated to meet the character restrictions and it came out sounding much more flippant. Sorry about that. It's a good question @RandomGuy. If you throw a trace around the matrix product the author's original comment makes more sense. I wonder if that's what he meant. – cardinal Mar 13 '11 at 15:21
    
Thank you for your answer. I see, it is fairly obvious once you understand the notation. If you convert your comment to an "answer", I'd be happy to accept it as the answer to this question. – RandomGuy Mar 20 '11 at 5:37
    
@cardinal +1 to make it a formal answer, I just found this thread when wondering about the very same thing. – alberto Jun 4 '15 at 6:55

It has been such a long time since you asked but I just want to give a proper answer with full equations.

$\begin{align*} p(x) &= \frac{1}{\sqrt{\left|2\pi \Sigma \right|}} \exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\log\left(\left|2\pi\Sigma \right| \right) \right\}\exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\left[\underbrace{x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x}_{\theta'T\left(x\right)} + \mu'\Sigma^{-1}\mu + \log \left(\left|2\pi\Sigma\right| \right)\right] \right\} \end{align*}$

To rearrange the original equation into the form of an exponential family, we need to use the relationship between Frobenius product and vectorizing operator.

$\begin{align*} x'\Sigma^{-1}x &= \Sigma^{-1}:xx'\\ &= \operatorname{vec}\left(\Sigma^{-1}\right)' \,\operatorname{vec}\left(xx' \right) \\ \mu' \Sigma^{-1} x &= \left(\Sigma^{-1}\mu \right)'x\end{align*}$

$\therefore x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x = \begin{bmatrix}\operatorname{vec}\left(\Sigma^{-1}\right) \\ -2\Sigma^{-1}\mu \end{bmatrix}'\begin{bmatrix}\operatorname{vec}\left(xx'\right) \\ x \end{bmatrix}$

share|cite|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.