1
$\begingroup$

Related questions here and here

I'm trying to compute the derivative of $S(X) = Tr(X\log(X))$. An additional condition is that $X = L^TL$ for some $L$ and I want to take the derivative with respect to $L$. So essentially, I would like to find $$\frac{\partial Tr\left(L^TL\log(L^TL)\right)}{\partial L}$$ This is well defined as it's the derivative of a scalar function with respect to a matrix so I should obtain a matrix of size $L$.

My progress so far:

Use the product rule to break it up into two derivatives. So I have $\frac{\partial Tr(L^T L C)}{\partial L}$, where $C = \log(L^TL)$ but is treated as a constant. According to the matrix cookbook, I get $L(C^T + C) = L(\log(L^TL))^T + L\log(L^TL)$.

The other term, $\frac{\partial Tr(C'\log(L^T L) )}{\partial L}$, is more troublesome. I thought about letting $L^TL = X$ and use the chain rule, but I'm not sure how to deal with the $\frac{\partial X}{\partial L}$ term since I now have the derivative of a matrix with respect to a matrix (instead of scalar with matrix).

So the question, after my work, is now how to do take the derivative of the second term. Alternatively, if the product rule was a bad idea and I can do it more directly, that would also be very helpful. Thank you.

$\endgroup$
2
$\begingroup$

Consider the following scalar function and its derivative. $$\eqalign{ f(x) &= x\,\log(x) \cr f'(x) &= 1 + \log(x) }$$ When the function is applied to a matrix argument the result is a matrix $$F = f(X)$$ The differential of the trace of $F$ is then $$\eqalign{ d\,{\rm Tr}(F) &= f'(X)^T:dX \cr &= \big(I+\log(X)\big)^T:dX \cr }$$ Now assume that $X$ is described in terms of another matrix $L$, i.e. $$\eqalign{ X &= L^TL \cr dX &= L^T\,dL + dL^T\,L \cr }$$ Substituting this into the previous result yields $$\eqalign{ d\,{\rm Tr}(F) &= \big(I+\log(L^TL)\big)^T:(L^T\,dL + dL^T\,L) \cr &= 2\,\big(I+\log(L^TL)\big):L^T\,dL \cr &= 2\,L\big(I+\log(L^TL)\big):dL \cr \frac{\partial{\,\rm Tr}(F)}{\partial L} &= 2\,\big(L+L\log(L^TL)\big) \cr\cr }$$ NB: In some steps above, a colon is used as a product notation for the trace, i.e. $$A:B = {\rm Tr}(A^TB)$$

$\endgroup$
  • $\begingroup$ Great answer! Just one question: How did you go from $\big(I+\log(L^TL)\big)^T:(L^T\,dL + dL^T\,L)$ to $2\big(I+\log(L^TL)\big):(L^T\,dL)$? Thank you $\endgroup$ – user1936752 Feb 23 at 20:49
  • 1
    $\begingroup$ It follows from these easily proved statements $$\eqalign{ (L^TdL)^T &= (L^TdL) \cr (\log(L^TL)+I)^T &= (\log(L^TL)+I) \cr A^T:B &= A:B^T \cr }$$ $\endgroup$ – greg Feb 23 at 23:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.