19
$\begingroup$

I'm a bit stumped by the exponential family representation of a multi-variate Gaussian distribution. Basically, the exponential form is a generic form for a large class of probability distributions. The standard form is

$$f_X(x) = \exp[\theta' T(x) + F(\theta)]$$

where $\theta$ is a set of parameters (based on $\mu$ and $\Sigma$), $T(x)$ is a vector of sufficient statistics, and $F$ is a function of the parameters that ensures the distribution is a pdf, i.e., sums to one. For more information on this form, see http://www.cs.columbia.edu/~jebara/4771/tutorials/lecture12.pdf, http://en.wikipedia.org/wiki/Exponential_family, etc.

The "conversion" for a multi-variate Gaussian distribution to exponential family form is listed as

$$\theta = [\Sigma^{-1}\mu, -\frac{1}{2}\Sigma^{-1}]'$$ $$T(x) = [x, x x']'$$

but this is confusing because the outer product $x x'$ is a matrix and $-\frac{1}{2}\Sigma^{-1}$ is also a matrix. Thus, it seems the product between $\theta$ and $T(x)$ should result in a scalar "entry" and a matrix "entry". Obviously, this expression needs to evaluate to a scalar.

The inner product works fine in the scalar case, and I understand this conversion is computed by manipulating to the quadratic form $x'Ax + b'x$. Still, it seems that I am completely missing something here. Thanks for your help.

$\endgroup$
6
  • 1
    $\begingroup$ Consider the "vectorized" version of $xx'$, then you'll see it matches the form. Or, just write it out in summation notation. Let $S = \Sigma^{-1}$ and $s_{ij}$ be the $(i,j)$th element of $S$. Define $y_{ij} = x_i x_j$. Let $u_i$ be the $i$th element of $S \mu$. Then the exponent of the density is $-\frac{1}{2} \sum_i \sum_j s_{ij} y_{ij} + \sum_i u_i x_i + h(S,u)$ which has the required form. You need to know that the the $y_{ij}$ and $x_i$ are linearly independent as well as the $s_{ij}$ and $u_i$ (so that you know you're not dealing with a curved exponential family. That's not hard. $\endgroup$
    – cardinal
    Mar 13, 2011 at 3:38
  • $\begingroup$ @cardinal Indeed, That's not hard, but @RandomGuy is right to point out there is at least an abuse of notation here, since what is written is a matrix product and what is meant is the scalar product of two $n\times n$ matrices transformed into vectors of size $n^2$ (as you aptly explain). If only every question on MSE could be as relevant as this one! $\endgroup$
    – Did
    Mar 13, 2011 at 10:12
  • $\begingroup$ @Didier, @RandomGuy, sorry I ran out of characters in my comment. My last sentence was supposed to say: "This last part is not hard to show." I truncated to meet the character restrictions and it came out sounding much more flippant. Sorry about that. It's a good question @RandomGuy. If you throw a trace around the matrix product the author's original comment makes more sense. I wonder if that's what he meant. $\endgroup$
    – cardinal
    Mar 13, 2011 at 15:21
  • 1
    $\begingroup$ @cardinal +1 to make it a formal answer, I just found this thread when wondering about the very same thing. $\endgroup$
    – alberto
    Jun 4, 2015 at 6:55
  • 1
    $\begingroup$ I ended up writing a blog post with a full derivation here $\endgroup$ Mar 11, 2020 at 16:57

1 Answer 1

22
$\begingroup$

It has been such a long time since you asked but I just want to give a proper answer with full equations.

$\begin{align*} p(x) &= \frac{1}{\sqrt{\left|2\pi \Sigma \right|}} \exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\log\left(\left|2\pi\Sigma \right| \right) \right\}\exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\left[\underbrace{x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x}_{\theta'T\left(x\right)} + \mu'\Sigma^{-1}\mu + \log \left(\left|2\pi\Sigma\right| \right)\right] \right\} \end{align*}$

To rearrange the original equation into the form of an exponential family, we need to use the relationship between Frobenius product and vectorizing operator.

$\begin{align*} x'\Sigma^{-1}x &= \Sigma^{-1}:xx'\\ &= \operatorname{vec}\left(\Sigma^{-1}\right)' \,\operatorname{vec}\left(xx' \right) \\ \mu' \Sigma^{-1} x &= \left(\Sigma^{-1}\mu \right)'x\end{align*}$

$\therefore x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x = \begin{bmatrix}\operatorname{vec}\left(\Sigma^{-1}\right) \\ -2\Sigma^{-1}\mu \end{bmatrix}'\begin{bmatrix}\operatorname{vec}\left(xx'\right) \\ x \end{bmatrix}$

$\endgroup$
7
  • 3
    $\begingroup$ Could you please add some more comments? I find this quite cryptic $\endgroup$ Jan 24, 2020 at 15:38
  • 3
    $\begingroup$ I ended up writing a blog post with a full derivation here $\endgroup$ Mar 11, 2020 at 16:57
  • $\begingroup$ This is very good. A small point, the normalizing constant in the first equation is to the power of n/2 for a MVN $\endgroup$
    – Allohvk
    Jan 28, 2022 at 8:39
  • 1
    $\begingroup$ @Allohvk $2\pi$ is inside the determinant. So no $\endgroup$
    – Daeyoung
    Jan 28, 2022 at 13:24
  • 1
    $\begingroup$ Thanks for the clarification. I will leave the comment for posterity in case anyone makes the same mistake as me $\endgroup$
    – Allohvk
    Jan 28, 2022 at 16:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .