1
$\begingroup$

I need help with taking the derivative of some quantities with respect to a diagonal matrix. Let say the diagonal matrix is $\boldsymbol{X}_d = \text{diag} \{ x_1, \dots, x_d \}$ (pardon my notation). I need to obtain the following derivatives $$\frac{\partial}{\partial \boldsymbol{X}_d} \text{Tr} \{ \boldsymbol{A} \boldsymbol{X}_d \boldsymbol{B} \} \quad \text{and} \quad \frac{\partial}{\partial \boldsymbol{X}_d} \ln | \boldsymbol{A} \boldsymbol{X}_d |.$$

Initially, I naively tried the formulas for the general matrix, but after I obtain the estimation of $\boldsymbol{X}_d$, I did not get a diagonal matrix, which does not make sense, so I know there must be something special with taking the derivative with respect to a diagonal matrix. I did not find many resources on this topic, so I want to post the question and get help. Please help me if you can. Thank you so much.

$\endgroup$

1 Answer 1

2
$\begingroup$

$\def\d{{\rm diag}}\def\D{{\rm Diag}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$Let's use a colon to denote the trace/Frobenius product $$\eqalign{ A:B &= {\rm Tr}(A^TB) \;=\; \sum_{j=1}^m\sum_{k=1}^n A_{jk} B_{jk} \\ A:A &= \big\|A\big\|_F^2 \\ }$$ In the case that $(A,B)$ are vectors, this definition corresponds to the standard dot product. The key idea is that the matrix/vector on each side of the colon must have the same dimensions.

The Frobenius product has many interesting properties.
In particular, for dimensionally compatible matrices $(A,B,C)$ and vector $(v)$ $$\eqalign{ AB:C &= A:CB^T \\&= B:A^TC \\&= C:AB \\ A:\D(v) &= \d(A):v \\ }$$ ${\bf NB}\!:\,$ The diag operator with an uppercase 'D' creates a diagonal matrix from a vector, while the one with the lowercase 'd' creates a vector from the diagonal of a matrix.

Write the first function in terms of this product.
Then calculate its differential and gradient. $$\eqalign{ \phi &= (BA):\D(x) \\&= \d(BA):x \\ d\phi &= \d(BA):dx \\ \p{\phi}{x} &= \d(BA) \\ \p{\phi}{X} &= \D\big(\d(BA)\big) = I\odot BA \\ }$$ where $\odot$ denotes the elementwise/Hadamard product and $I$ is the identity matrix.

For the second function, let $\,Y=AX\;$ and use Jacobi's formula $$\eqalign{ \psi &= \log(\det(Y)) \\ d\psi &= Y^{-T}:dY \\ &= (AX)^{-T}:A\,dX \\ &= A^T(A^{-T}X^{-T}):dX \\ &= X^{-1}:\D(dx) \\ &= \d(X^{-1}):dx \\ \p{\psi}{x} &= \d(X^{-1}) \\ \p{\psi}{X} &= \D\big(\d(X^{-1})\big) = I\odot X^{-1} = X^{-1} \\ }$$ since $X\,\left({\rm and}\,X^{-1}\right)$ is a already a diagonal matrix the Diag operator has no effect.

$\endgroup$
2
  • $\begingroup$ Thank you so much for your explanation. It is very helpful. May I ask if you are aware of any books about matrix calculus? I've found myself using matrix calculus a lot recently. I mainly use the matrix cookbook for reference without much understanding of what goes behind. $\endgroup$
    – little_sky
    Apr 7, 2021 at 22:01
  • 1
    $\begingroup$ If you want more formulas, then in addition to The Matrix Cookbook, I'd suggest Bernstein's Matrix Mathematics: Theory, Facts, and Formulas. If you want more theory, then Magnus and Neudecker's Matrix Differential Calculus, or perhaps Hjorungnes's Complex-Valued Matrix Derivatives, or just poke around on this site $-$ there are lots of interesting questions and answers. $\endgroup$
    – greg
    Apr 8, 2021 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.