0
$\begingroup$

Let $D$ be positive diagonal and $W$ be symmetric positive definite (spd). According to relation $124$ in matrix cookbook, the derivative of tr$(WD^{-1})$ with respect to $D$ is $-D^{-1}WD^{-1}$.

How do we solve for the derivative of tr$(WD^{-2})$ with respect to $D$? I tried the Frobenius product approach, but couldn't work the steps.

$\endgroup$
  • $\begingroup$ are you familiar with the Frechet Derivative? $\endgroup$ – peek-a-boo May 28 at 2:03
  • $\begingroup$ No, but will take a look. Thanks. $\endgroup$ – Kay May 28 at 3:21
2
$\begingroup$

For convenience, define the diagonal matrix $\,V= {\rm Diag}(W) = I\odot W$

Diagonal matrices are easy to work with because they're symmetric and they commute with each other, e.g. $\,DV=VD=V^TD$

Write the function in terms of this new variable and the Frobenius product. Then find its differential and gradient. $$\eqalign{ \phi &= {\rm Tr}(VD^{-2}) \cr&= V:D^{-2} \cr d\phi &= V:(-2D^{-3}dD) \cr&= -2D^{-3}V:dD \cr \frac{\partial\phi}{\partial D} &= -2D^{-3}V \cr }$$ Note that ${\rm Tr}(VD^k) = {\rm Tr}(WD^k)$, i.e. the off-diagonal components of $W$ contribute nothing.

$\endgroup$
  • $\begingroup$ Thanks @greg. However, I have the following concern. Similar to the steps you provided, $$\eqalign{ Let \ \psi &= {\rm Tr}(VD^{-1}) \cr&= V:D^{-1} \cr d\psi &= V:(-D^{-2}dD) \cr&= -D^{-2}V:dD \cr \frac{\partial\psi}{\partial D} &= -D^{-2}V = -D^{-1}VD^{-1} \cr }$$ Meanwhile, the derivative of tr$(WD^{-1})$ with respect to $D$ is $-D^{-1}WD^{-1} \ $([matrix cookbook][1]). My concern being the inequality $-D^{-1}VD^{-1} \ne -D^{-1}WD^{-1}$. [1]: math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf $\endgroup$ – Kay May 28 at 3:41
  • $\begingroup$ First, since we're dealing with commuting (diagonal) matrices the cookbook result can be simplified to $D^{-1}VD^{-1}=D^{-2}V$. Second, the gradient of a scalar with respect to a diagonal matrix should be another diagonal matrix. Think about it, what would give rise to an off-diagonal element? $\endgroup$ – greg May 28 at 3:58
  • $\begingroup$ Got it! "the gradient of a scalar with respect to a diagonal matrix should be another diagonal matrix". I appreciate Greg. $\endgroup$ – Kay May 28 at 4:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.