# Taking derivative with respect to a diagonal matrix

I need help with taking the derivative of some quantities with respect to a diagonal matrix. Let say the diagonal matrix is $$\boldsymbol{X}_d = \text{diag} \{ x_1, \dots, x_d \}$$ (pardon my notation). I need to obtain the following derivatives $$\frac{\partial}{\partial \boldsymbol{X}_d} \text{Tr} \{ \boldsymbol{A} \boldsymbol{X}_d \boldsymbol{B} \} \quad \text{and} \quad \frac{\partial}{\partial \boldsymbol{X}_d} \ln | \boldsymbol{A} \boldsymbol{X}_d |.$$

Initially, I naively tried the formulas for the general matrix, but after I obtain the estimation of $$\boldsymbol{X}_d$$, I did not get a diagonal matrix, which does not make sense, so I know there must be something special with taking the derivative with respect to a diagonal matrix. I did not find many resources on this topic, so I want to post the question and get help. Please help me if you can. Thank you so much.

$$\def\d{{\rm diag}}\def\D{{\rm Diag}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$$Let's use a colon to denote the trace/Frobenius product \eqalign{ A:B &= {\rm Tr}(A^TB) \;=\; \sum_{j=1}^m\sum_{k=1}^n A_{jk} B_{jk} \\ A:A &= \big\|A\big\|_F^2 \\ } In the case that $$(A,B)$$ are vectors, this definition corresponds to the standard dot product. The key idea is that the matrix/vector on each side of the colon must have the same dimensions.

The Frobenius product has many interesting properties.
In particular, for dimensionally compatible matrices $$(A,B,C)$$ and vector $$(v)$$ \eqalign{ AB:C &= A:CB^T \\&= B:A^TC \\&= C:AB \\ A:\D(v) &= \d(A):v \\ } $${\bf NB}\!:\,$$ The diag operator with an uppercase 'D' creates a diagonal matrix from a vector, while the one with the lowercase 'd' creates a vector from the diagonal of a matrix.

Write the first function in terms of this product.
Then calculate its differential and gradient. \eqalign{ \phi &= (BA):\D(x) \\&= \d(BA):x \\ d\phi &= \d(BA):dx \\ \p{\phi}{x} &= \d(BA) \\ \p{\phi}{X} &= \D\big(\d(BA)\big) = I\odot BA \\ } where $$\odot$$ denotes the elementwise/Hadamard product and $$I$$ is the identity matrix.

For the second function, let $$\,Y=AX\;$$ and use Jacobi's formula \eqalign{ \psi &= \log(\det(Y)) \\ d\psi &= Y^{-T}:dY \\ &= (AX)^{-T}:A\,dX \\ &= A^T(A^{-T}X^{-T}):dX \\ &= X^{-1}:\D(dx) \\ &= \d(X^{-1}):dx \\ \p{\psi}{x} &= \d(X^{-1}) \\ \p{\psi}{X} &= \D\big(\d(X^{-1})\big) = I\odot X^{-1} = X^{-1} \\ } since $$X\,\left({\rm and}\,X^{-1}\right)$$ is a already a diagonal matrix the Diag operator has no effect.

• Thank you so much for your explanation. It is very helpful. May I ask if you are aware of any books about matrix calculus? I've found myself using matrix calculus a lot recently. I mainly use the matrix cookbook for reference without much understanding of what goes behind. Apr 7, 2021 at 22:01
• If you want more formulas, then in addition to The Matrix Cookbook, I'd suggest Bernstein's Matrix Mathematics: Theory, Facts, and Formulas. If you want more theory, then Magnus and Neudecker's Matrix Differential Calculus, or perhaps Hjorungnes's Complex-Valued Matrix Derivatives, or just poke around on this site $-$ there are lots of interesting questions and answers.
– greg
Apr 8, 2021 at 14:51