# What is the dot product between a vector of matrices?

There is a notation used in many sources (e.g. Wikipedia: http://en.wikipedia.org/wiki/Exponential_family) for the natural parameters of exponential family distributions which I do not understand, and I cannot find a description of.

With vector parameters and variables, the exponential family form has the dot product between the vector natural parameter, ${\boldsymbol\eta}({\boldsymbol\theta})$ and the vector sufficient statistic, ${\mathbf{T}}({\mathbf{x}})$, in the exponent. i.e. $e^{{\boldsymbol\eta}({\boldsymbol\theta})^{\top}{\mathbf{T}}({\mathbf{x}})}$.

However, many examples of these parameters for different distributions are vectors composed of matrices & vectors. E.g. the multivariate Normal distribution has parameter $[\Sigma^{-1}\mu\space\space-\frac{1}{2}\Sigma^{-1}]$ and sufficient statistic $[\mathbf{x}\space\space\mathbf{xx^{\top}}]$.

So what are these "vectors" and moreover, how is the dot product between them defined? Does this notation have a name?

I believe you're supposed to "vectorize" the matrix, i.e. rearrange into a $n^2 \times 1$ vector.

Equivalently, you can take $A\cdot B = \textrm{tr}(A^TB)$ as the definition.

EDIT - Example:

If $A = \left(\begin{array}{cc}a & b \\ c & d \end{array}\right)$, $B = \left(\begin{array}{cc}e & f \\ g & h \end{array}\right)$, then

$$A^TB = \left(\begin{array}{cc}ae + cg & af + ch \\ be + dg & bf + dh \end{array}\right)$$

and the trace is $ae + bf + cg + dh$. Likewise, if we first vectorize the matrices $$\widetilde{A} = \left(\begin{array}{c}a & b & c & d \end{array}\right)^T\\ \widetilde{B} = \left(\begin{array}{c}e & f & g & h \end{array}\right)^T\\$$

it's straightforward to see $\widetilde{A}\cdot\widetilde{B} = \textrm{tr}(A^TB)$.

• I see: vectorizing each element makes sense. However, I don't see how using the trace definition helps since the first element of each vector is a vector, while the second element of each is a matrix. – Chris Burgess Oct 10 '13 at 14:33
• The result in either case is a scalar though. I'll add a quick example to my answer. – BaronVT Oct 10 '13 at 15:06
• Oh, I see what you're saying. If the first entry of the outer matrix is a vector (n x 1), and the second entry is an inner matrix (n x n), then the outer matrix is (n x (n + 1)) and this will still work. (after transposing the first matrix, you'll be multiplying an ((n+1) x n) matrix by an (n x (n+1)) one, the result is ((n+1) x (n+1)), and then you take the trace.) – BaronVT Oct 10 '13 at 15:20
• Ah I see. So I can think of the two outer vectors as essentially defining a pair of block matrices (in each, a column plus a matrix). Then I can use the trace definition and it works equivalently to vectorization. Thanks – Chris Burgess Oct 10 '13 at 16:33