There is a notation used in many sources (e.g. Wikipedia: http://en.wikipedia.org/wiki/Exponential_family) for the natural parameters of exponential family distributions which I do not understand, and I cannot find a description of.

With vector parameters and variables, the exponential family form has the dot product between the vector natural parameter, ${\boldsymbol\eta}({\boldsymbol\theta})$ and the vector sufficient statistic, ${\mathbf{T}}({\mathbf{x}})$, in the exponent. i.e. $e^{{\boldsymbol\eta}({\boldsymbol\theta})^{\top}{\mathbf{T}}({\mathbf{x}})}$.

However, many examples of these parameters for different distributions are vectors composed of matrices & vectors. E.g. the multivariate Normal distribution has parameter $[\Sigma^{-1}\mu\space\space-\frac{1}{2}\Sigma^{-1}]$ and sufficient statistic $[\mathbf{x}\space\space\mathbf{xx^{\top}}]$.

So what are these "vectors" and moreover, how is the dot product between them defined? Does this notation have a name?


I believe you're supposed to "vectorize" the matrix, i.e. rearrange into a $n^2 \times 1$ vector.

Equivalently, you can take $A\cdot B = \textrm{tr}(A^TB)$ as the definition.

EDIT - Example:

If $A = \left(\begin{array}{cc}a & b \\ c & d \end{array}\right)$, $B = \left(\begin{array}{cc}e & f \\ g & h \end{array}\right)$, then

$$ A^TB = \left(\begin{array}{cc}ae + cg & af + ch \\ be + dg & bf + dh \end{array}\right) $$

and the trace is $ae + bf + cg + dh$. Likewise, if we first vectorize the matrices $$ \widetilde{A} = \left(\begin{array}{c}a & b & c & d \end{array}\right)^T\\ \widetilde{B} = \left(\begin{array}{c}e & f & g & h \end{array}\right)^T\\ $$

it's straightforward to see $\widetilde{A}\cdot\widetilde{B} = \textrm{tr}(A^TB)$.

  • $\begingroup$ I see: vectorizing each element makes sense. However, I don't see how using the trace definition helps since the first element of each vector is a vector, while the second element of each is a matrix. $\endgroup$ – Chris Burgess Oct 10 '13 at 14:33
  • $\begingroup$ The result in either case is a scalar though. I'll add a quick example to my answer. $\endgroup$ – BaronVT Oct 10 '13 at 15:06
  • $\begingroup$ Oh, I see what you're saying. If the first entry of the outer matrix is a vector (n x 1), and the second entry is an inner matrix (n x n), then the outer matrix is (n x (n + 1)) and this will still work. (after transposing the first matrix, you'll be multiplying an ((n+1) x n) matrix by an (n x (n+1)) one, the result is ((n+1) x (n+1)), and then you take the trace.) $\endgroup$ – BaronVT Oct 10 '13 at 15:20
  • $\begingroup$ Ah I see. So I can think of the two outer vectors as essentially defining a pair of block matrices (in each, a column plus a matrix). Then I can use the trace definition and it works equivalently to vectorization. Thanks $\endgroup$ – Chris Burgess Oct 10 '13 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.