derivative of exponential of matrix trace

What is the derivative of $\sum_{ij}e^{-d_{ij}^2(X)}=\sum_{ij}e^{-\operatorname{tr}(X^TC_{ij}X)}$, w.r.t $X$ where $C_{ij}$ is a constant matrix and $d_{ij}^2(X)$ denotes the squared Euclidean distance between the rows $i,j$ of $X$. All the entries here are real

-
You need to clarify your notation a bit. Is $C_{ij}$ a constant matrix for each pair $ij$, i.e. a family of matrices labeled by two indices, $i$ and $j$, or is $C$ a matrix with components $C_{ij}$? If the latter, then there is no additional summation over $i$ and $j$ after evaluating $e^{-{\rm tr}(X^T C X)}$. – josh Jun 5 '13 at 3:45
@josh it is a family of matrices labeled by $i,j$. In fact it is the matrix $C_{ij}$ formed by $(e_i-e_j)(e_i-e_j)^T$ where the $e's$ are the basis vectors. – user75402 Jun 5 '13 at 3:47

3 Answers

Okay. It doesn't change much anyhow. Use linearity of the trace. Writing $f(X) = {\rm tr}(X^T C_{ij} X)$ and varying $X$ by $\delta X$, we get $f(X+\delta X) - f(X) = {\rm tr}(\delta X^T C_{ij} X) + {\rm tr}(X^T C_{ij} \delta X)$. Now use what you know about how matrix traces transform under transposition of the argument and also what you know about the form of $C_{ij}$ to simplify that expression and then give the matrix derivative of $g(X)$.

What about the derivative of $g(X) = \exp f(X)$? Since $f$ maps vectors to real numbers, you can use the familiar composition rule on the exponentiation.

You may find that your expression of $C_{ij}$ pulls out components of $X$. What does the final summation over $i$ and $j$ do?

-
So is it, $-2\sum_{ij}(e^{-tr X^TC_{ij}X})C_{ij}X$ ? If not, please correct it. – user75402 Jun 5 '13 at 4:50
Waiting for a definitive answer.. – user75402 Jun 8 '13 at 0:04

User, the image of the derivative is a scalar ! Assume that the matrices are real. Moreover, the $(C_{i,j})$ are symmetric matrices. Then the required derivative is $H\rightarrow -2\sum_{i,j}Trace(X^TC_{i,j}H)exp(-Trace(X^TC_{i,j}X))$.

-

Consider the scalar function \eqalign{ f_{ij} &= {\rm exp}(-C_{ij}:XX^T)\cr } Your objective function is simply the sum of these functions: $\,\,f=\sum_{ij}f_{ij}$

Next, consider the differential of the logarithm of one of these scalar functions \eqalign{ {\rm log}(f_{ij}) &= -C_{ij}:XX^T \cr d\,{\rm log}(f_{ij}) &= -C_{ij}:d\,(XX^T) \cr \frac {df_{ij}}{f_{ij}} &= -C_{ij}:d\,(XX^T) \cr &= -2\,C_{ij}\,X:dX \cr df_{ij} &= -2\,f_{ij}\,C_{ij}\,X:dX \cr \frac {\partial f_{ij}}{\partial X} &= -2\,f_{ij}\,C_{ij}\,X \cr } The derivative of the objective function is the sum of these derivatives \eqalign{ \frac {\partial f}{\partial X} &= -2\,\sum_{ij} \,f_{ij}\,C_{ij}\,X \cr } You can sum the indexed quantities and collect them into a single matrix $M = \sum_{ij} \,f_{ij}\,C_{ij}$. Now you can write the derivative as $\,\,\frac {\partial f}{\partial X} = -2MX$

-