# Partial derivative of the trace of matrix entry-wise exponential?

Just checking my math here and getting some help for the exponential part.


We have that $\trace{\mathbf F(\mathbf X)} = \v{\mathbf F(\mathbf X)}^\top \v{\mathbf I}$, so its derivative is $\v{\partial \m F(\m X)}^\top\v{\m I}$.

We also have $\partial \exp f(x) / \partial x = \exp f(x)\partial f(x)/\partial x$.

I guess the partial derivative $\partial \exp \m X \m X^\top / \partial \m X$ must be something like $$\diag{\v{\exp \m X \m X^\top}}(\m I \otimes \m X+\m X \boxtimes \m I)$$

So in the end, the derivative I'm looking for must be equal to $2\m X^\top\exp(\m X\m X^\top)$ ?

The final expression is correct in the sense that if $G:X\mapsto\mathrm{tr}(\exp(XX^T))$, then $$\langle\nabla G(X),Z\rangle=2\mathrm{tr}(ZX^T\exp(XX^T))=2\mathrm{tr}(\exp(XX^T)XZ^T).$$ Thus, if one identifies the gradient $\nabla G(X)$ with the matrix $M_X$ such that $$\langle\nabla G(X),Z\rangle=\mathrm{tr}(ZM_X),$$ then indeed, the gradient is $$2X^T\exp(XX^T)=2\exp(X^TX)X^T.$$ A more logical convention would be to identify the gradient $\nabla G(X)$ with the matrix $M_X$ such that $$\langle\nabla G(X),Z\rangle=\mathrm{tr}(ZM_X^T)=\mathrm{tr}(MZ_X^T),$$ then the gradient would rather be $$2\exp(XX^T)X=2X\exp(X^TX).$$ The intermediate steps proposed in the question are opaque to me. A direct proof of the result uses simply the characterization of the differential $\nabla G(X)$ at $X$ as the unique linear functional on the space of matrices of size $n\times m$ such that, for every such matrix $Z$ and real number $h$, when $h\to0$, $$G(X+hZ)=G(X)+\langle\nabla G(X),Z\rangle h+o(h).$$
• What is stopping you when you try to find some $L$ such that $G(X+hZ)=G(X)+hL+o(h)$? – Did Oct 3 '14 at 13:54
• I know $o(h)$ is supposed to tend to $0$. But, written like that, it looks like it should only depend on $h$. – davcha Oct 6 '14 at 7:46
• ?? $o(h)$ is there to denote any term $A(h)$ such that $A(h)/h\to0$, not only $A(h)\to0$. Can you give me more details about your methods to compute derivatives, in general? – Did Oct 6 '14 at 7:48
• Right, I forgot about $A(h)/h$. Usually, I do something like this, for example : $\partial trace(XX^\top) / \partial X = (trace(\partial X X^\top) + trace(X\partial X^\top))/\partial X$, then I manipulate the formula to take away $\partial X$. – davcha Oct 6 '14 at 8:00
The differential of the trace of a scalar function ($f$) with derivative ($f'$) applied to a square matrix argument ($Y$) is \eqalign { d\,{\rm tr}(f(Y)) &= f'(Y^T):dY \cr } The exponential function is its own derivative, so $f'=f$ and \eqalign { d\,{\rm tr}({\rm exp}(Y)) &= {\rm exp}(Y^T):dY \cr } In the case that $Y=(XX^T)$, we have $Y=Y^T$ and \eqalign { d\,{\rm tr}({\rm exp}(Y)) &= {\rm exp}(Y):(dX\,X^T + X\,dX^T) \cr &= {\rm exp}(Y):2\,{\rm sym}(dX\,X^T) \cr &= 2\,{\rm sym}({\rm exp}(Y)):dX\,X^T \cr &= 2\,{\rm exp}(Y)\,X:dX \cr &= 2\,{\rm exp}(XX^T)\,X:dX \cr } The derivative is \eqalign { \frac {\partial\,{\rm tr}({\rm exp}(Y))} {\partial X} &= 2\,{\rm exp}(XX^T)\,X \cr &= 2\,X\,{\rm exp}(X^TX)\ \cr } Where the final equality follows from this general property of matrix functions \eqalign { f(AB^T)\,A &= A\,f(B^TA) \cr } where $A,B \in {\mathbb R}^{m\times n}$