# Replace $X$ with $\mbox{diag}(x)$ in trace matrix derivative identity

There is a scaler-by-matrix derivative identity:

$$\frac{\partial}{\partial X}trace\left(AXBX'C\right)=B'X'A'C'+BX'CA$$

How does this change if instead I am trying to find

$$\frac{\partial}{\partial x}trace\left(Adiag(x)Bdiag(x)'C\right)$$

where $x$ is a vector rather than a matrix.

My thinking is that all I have to do is multiply the original identity by a vector of ones as that would be the derivative of $diag(x)$. However, I'm not sure how the chain rule interacts with traces.

I ask as I am trying to calculate. $$\frac{\partial}{\partial w}trace\left(Ddiag(w)\Omega diag(w)D'\right)$$

where $w \mathbb{\in R^{N}}$, $D\mathbb{\in R^{M\times N}}$, and $\Omega\mathbb{\in R^{N\times N}}$. Also $\Omega$ can be assumed to be positive definite.

This implies the result would be

$$\left(2\Omega diag(w)D'D\right)e$$

where $e \mathbb{\in R^{N}}$ is a vector of ones.

• Don't multiply the matrix by $e$, but create a vector from its diagonal elements $${\rm diag}\Big(2\Omega\,\,{\rm Diag}(w)\,D'D\Big)$$ – greg Jan 19 '18 at 3:44
• @greg I appreciate this. Last night I had manually calculated the derivative when $w$ was length 5. When I got around to putting it in the computer today, I was able to verify that it was equivalent to your result. I would be happy to accept yours as the answer if you put it as one. – John Jan 19 '18 at 16:21

Let $f : \mathbb R^n \to \mathbb R$ be defined by

$$f (\mathrm x) := \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \right)$$

where $\mathrm A \in \mathbb R^{m \times n}$, $\mathrm B \in \mathbb R^{n \times n}$ and $\mathrm C \in \mathbb R^{n \times m}$ are given. The directional derivative of $f$ in the direction of $\mathrm v \in \mathbb R^n$ at $\mathrm x \in \mathbb R^n$ is given by

$$\begin{array}{rl} \displaystyle\lim_{h \to 0} \dfrac{f (\mathrm x + h \,\mathrm v) - f (\mathrm x)}{h} &= \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm v) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \right) + \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \, \mbox{diag} (\mathrm v) \, \mathrm C \right)\\ &= \mbox{tr} \left( \mbox{diag} (\mathrm v) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mbox{tr} \left( \mbox{diag} (\mathrm v) \, \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)\\ &= \mathrm v^\top \mbox{diag}^{-1} \left( \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mathrm v^\top \mbox{diag}^{-1} \left( \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)\end{array}$$

where $\mbox{diag}^{-1} : \mathbb R^{n \times n} \to \mathbb R^n$ is a linear function that takes a square matrix and extracts its main diagonal as a column vector. Thus, the gradient of $f$ is

$$\nabla_{\mathrm x} f(\mathrm x) = \color{blue}{\mbox{diag}^{-1} \left( \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mbox{diag}^{-1} \left( \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)}$$

• BTW, I really like the notation for inverse diagonal that you used. – John Jan 22 '18 at 17:40
• I didn't invent it. Take a look at page 10 of this. – Rodrigo de Azevedo Jan 22 '18 at 19:30

$$\def\v{{\rm vec}}\def\d{{\rm diag}}\def\D{{\rm Diag}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$$For typing convenience, use a colon as a product notation for the trace, i.e. \eqalign{ A:B = {\rm Tr}(AB^T) \;=\; \sum_{i=1}^m \sum_{j=1}^n A_{ij} B_{ij} \\ } and assign a name to the function of interest \eqalign{ \phi &= {\rm Tr}\left(AXBX^TC\right) \\ &= CAXB:X \\&= A^TC^TXB^T:X \\ } Then the gradient that you discovered can be written as the differential relationship \eqalign{ d\phi &= \big(CAXB + A^TC^TXB^T\big):dX \\ } Let's also carefully name the diagonal operations. The diag() function creates a vector from the diagonal of its matrix argument, while the Diag() function does the opposite - creating a diagonal matrix from a vector argument, e.g. \eqalign{ X = \D(x) \quad\implies\quad x = \d(X) \\ } The colon product has a very interesting property with respect to these operators \eqalign{ A:\D(x) &= \d(A):x \\ } Using all of the above, we can calculate the gradient of interest as follows \eqalign{ d\phi &= \big(CAXB + A^TC^TXB^T\big):\D(dx) \\ &= \d\big(CAXB + A^TC^TXB^T\big):dx \\ \p{\phi}{x} ​&= \d\big(CAXB + A^TC^TXB^T\big) \\ } Substituting $$A=D,\,C=D^T\,$$ and $$B=\Omega=\Omega^T,\,$$ the gradient can be simplified to \eqalign{ \p{\phi}{x} ​&= \d\big(D^TDX\Omega + D^TDX\Omega\big) \\ ​&= \d\big(2\,D^TDX\Omega\big) \\ } Since $$\d(A^T)=\d(A)$$, this can also be written as \eqalign{ \p{\phi}{x} ​&= \d\big(2\,\Omega XD^TD\big) \qquad\qquad \\ }