# Exponential family representation of multi-variate Gaussians

I'm a bit stumped by the exponential family representation of a multi-variate Gaussian distribution. Basically, the exponential form is a generic form for a large class of probability distributions. The standard form is

$$f_X(x) = \exp[\theta' T(x) + F(\theta)]$$

where $\theta$ is a set of parameters (based on $\mu$ and $\Sigma$), $T(x)$ is a vector of sufficient statistics, and $F$ is a function of the parameters that ensures the distribution is a pdf, i.e., sums to one. For more information on this form, see http://www.cs.columbia.edu/~jebara/4771/tutorials/lecture12.pdf, http://en.wikipedia.org/wiki/Exponential_family, etc.

The "conversion" for a multi-variate Gaussian distribution to exponential family form is listed as

$$\theta = [\Sigma^{-1}\mu, -\frac{1}{2}\Sigma^{-1}]'$$ $$T(x) = [x, x x']'$$

but this is confusing because the outer product $x x'$ is a matrix and $-\frac{1}{2}\Sigma^{-1}$ is also a matrix. Thus, it seems the product between $\theta$ and $T(x)$ should result in a scalar "entry" and a matrix "entry". Obviously, this expression needs to evaluate to a scalar.

The inner product works fine in the scalar case, and I understand this conversion is computed by manipulating to the quadratic form $x'Ax + b'x$. Still, it seems that I am completely missing something here. Thanks for your help.

• Consider the "vectorized" version of $xx'$, then you'll see it matches the form. Or, just write it out in summation notation. Let $S = \Sigma^{-1}$ and $s_{ij}$ be the $(i,j)$th element of $S$. Define $y_{ij} = x_i x_j$. Let $u_i$ be the $i$th element of $S \mu$. Then the exponent of the density is $-\frac{1}{2} \sum_i \sum_j s_{ij} y_{ij} + \sum_i u_i x_i + h(S,u)$ which has the required form. You need to know that the the $y_{ij}$ and $x_i$ are linearly independent as well as the $s_{ij}$ and $u_i$ (so that you know you're not dealing with a curved exponential family. That's not hard. Mar 13, 2011 at 3:38
• @cardinal Indeed, That's not hard, but @RandomGuy is right to point out there is at least an abuse of notation here, since what is written is a matrix product and what is meant is the scalar product of two $n\times n$ matrices transformed into vectors of size $n^2$ (as you aptly explain). If only every question on MSE could be as relevant as this one!
– Did
Mar 13, 2011 at 10:12
• @Didier, @RandomGuy, sorry I ran out of characters in my comment. My last sentence was supposed to say: "This last part is not hard to show." I truncated to meet the character restrictions and it came out sounding much more flippant. Sorry about that. It's a good question @RandomGuy. If you throw a trace around the matrix product the author's original comment makes more sense. I wonder if that's what he meant. Mar 13, 2011 at 15:21
• @cardinal +1 to make it a formal answer, I just found this thread when wondering about the very same thing. Jun 4, 2015 at 6:55
• I ended up writing a blog post with a full derivation here Mar 11, 2020 at 16:57

It has been such a long time since you asked but I just want to give a proper answer with full equations.

\begin{align*} p(x) &= \frac{1}{\sqrt{\left|2\pi \Sigma \right|}} \exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\log\left(\left|2\pi\Sigma \right| \right) \right\}\exp \left\{-\frac{1}{2}\left(x-\mu\right)'\Sigma^{-1}\left(x-\mu \right) \right\}\\ &= \exp \left\{-\frac{1}{2}\left[\underbrace{x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x}_{\theta'T\left(x\right)} + \mu'\Sigma^{-1}\mu + \log \left(\left|2\pi\Sigma\right| \right)\right] \right\} \end{align*}

To rearrange the original equation into the form of an exponential family, we need to use the relationship between Frobenius product and vectorizing operator.

\begin{align*} x'\Sigma^{-1}x &= \Sigma^{-1}:xx'\\ &= \operatorname{vec}\left(\Sigma^{-1}\right)' \,\operatorname{vec}\left(xx' \right) \\ \mu' \Sigma^{-1} x &= \left(\Sigma^{-1}\mu \right)'x\end{align*}

$\therefore x'\Sigma^{-1}x - 2\mu'\Sigma^{-1}x = \begin{bmatrix}\operatorname{vec}\left(\Sigma^{-1}\right) \\ -2\Sigma^{-1}\mu \end{bmatrix}'\begin{bmatrix}\operatorname{vec}\left(xx'\right) \\ x \end{bmatrix}$

• Could you please add some more comments? I find this quite cryptic Jan 24, 2020 at 15:38
• I ended up writing a blog post with a full derivation here Mar 11, 2020 at 16:57
• This is very good. A small point, the normalizing constant in the first equation is to the power of n/2 for a MVN Jan 28, 2022 at 8:39
• @Allohvk $2\pi$ is inside the determinant. So no Jan 28, 2022 at 13:24
• Thanks for the clarification. I will leave the comment for posterity in case anyone makes the same mistake as me Jan 28, 2022 at 16:48