There is a scaler-by-matrix derivative identity:

$$\frac{\partial}{\partial X}trace\left(AXBX'C\right)=B'X'A'C'+BX'CA$$

How does this change if instead I am trying to find

$$\frac{\partial}{\partial x}trace\left(Adiag(x)Bdiag(x)'C\right)$$

where $x$ is a vector rather than a matrix.

My thinking is that all I have to do is multiply the original identity by a vector of ones as that would be the derivative of $diag(x)$. However, I'm not sure how the chain rule interacts with traces.

I ask as I am trying to calculate. $$\frac{\partial}{\partial w}trace\left(Ddiag(w)\Omega diag(w)D'\right)$$

where $w \mathbb{\in R^{N}}$, $D\mathbb{\in R^{M\times N}}$, and $\Omega\mathbb{\in R^{N\times N}}$. Also $\Omega$ can be assumed to be positive definite.

This implies the result would be

$$\left(2\Omega diag(w)D'D\right)e$$

where $e \mathbb{\in R^{N}}$ is a vector of ones.

  • $\begingroup$ Don't multiply the matrix by $e$, but create a vector from its diagonal elements $${\rm diag}\Big(2\Omega\,\,{\rm Diag}(w)\,D'D\Big)$$ $\endgroup$ – greg Jan 19 '18 at 3:44
  • $\begingroup$ @greg I appreciate this. Last night I had manually calculated the derivative when $w$ was length 5. When I got around to putting it in the computer today, I was able to verify that it was equivalent to your result. I would be happy to accept yours as the answer if you put it as one. $\endgroup$ – John Jan 19 '18 at 16:21

Let $f : \mathbb R^n \to \mathbb R$ be defined by

$$f (\mathrm x) := \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \right)$$

where $\mathrm A \in \mathbb R^{m \times n}$, $\mathrm B \in \mathbb R^{n \times n}$ and $\mathrm C \in \mathbb R^{n \times m}$ are given. The directional derivative of $f$ in the direction of $\mathrm v \in \mathbb R^n$ at $\mathrm x \in \mathbb R^n$ is given by

$$\begin{array}{rl} \displaystyle\lim_{h \to 0} \dfrac{f (\mathrm x + h \,\mathrm v) - f (\mathrm x)}{h} &= \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm v) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \right) + \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \, \mbox{diag} (\mathrm v) \, \mathrm C \right)\\ &= \mbox{tr} \left( \mbox{diag} (\mathrm v) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mbox{tr} \left( \mbox{diag} (\mathrm v) \, \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)\\ &= \mathrm v^\top \mbox{diag}^{-1} \left( \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mathrm v^\top \mbox{diag}^{-1} \left( \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)\end{array}$$

where $\mbox{diag}^{-1} : \mathbb R^{n \times n} \to \mathbb R^n$ is a linear function that takes a square matrix and extracts its main diagonal as a column vector. Thus, the gradient of $f$ is

$$\nabla_{\mathrm x} f(\mathrm x) = \color{blue}{\mbox{diag}^{-1} \left( \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mbox{diag}^{-1} \left( \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)}$$

  • $\begingroup$ BTW, I really like the notation for inverse diagonal that you used. $\endgroup$ – John Jan 22 '18 at 17:40
  • 1
    $\begingroup$ I didn't invent it. Take a look at page 10 of this. $\endgroup$ – Rodrigo de Azevedo Jan 22 '18 at 19:30

$\def\v{{\rm vec}}\def\d{{\rm diag}}\def\D{{\rm Diag}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$For typing convenience, use a colon as a product notation for the trace, i.e. $$\eqalign{ A:B = {\rm Tr}(AB^T) \;=\; \sum_{i=1}^m \sum_{j=1}^n A_{ij} B_{ij} \\ }$$ and assign a name to the function of interest $$\eqalign{ \phi &= {\rm Tr}\left(AXBX^TC\right) \\ &= CAXB:X \\&= A^TC^TXB^T:X \\ }$$ Then the gradient that you discovered can be written as the differential relationship $$\eqalign{ d\phi &= \big(CAXB + A^TC^TXB^T\big):dX \\ }$$ Let's also carefully name the diagonal operations. The diag() function creates a vector from the diagonal of its matrix argument, while the Diag() function does the opposite - creating a diagonal matrix from a vector argument, e.g. $$\eqalign{ X = \D(x) \quad\implies\quad x = \d(X) \\ }$$ The colon product has a very interesting property with respect to these operators $$\eqalign{ A:\D(x) &= \d(A):x \\ }$$ Using all of the above, we can calculate the gradient of interest as follows $$\eqalign{ d\phi &= \big(CAXB + A^TC^TXB^T\big):\D(dx) \\ &= \d\big(CAXB + A^TC^TXB^T\big):dx \\ \p{\phi}{x} ​&= \d\big(CAXB + A^TC^TXB^T\big) \\ }$$ Substituting $A=D,\,C=D^T\,$ and $B=\Omega=\Omega^T,\,$ the gradient can be simplified to $$\eqalign{ \p{\phi}{x} ​&= \d\big(D^TDX\Omega + D^TDX\Omega\big) \\ ​&= \d\big(2\,D^TDX\Omega\big) \\ }$$ Since $\d(A^T)=\d(A)$, this can also be written as $$\eqalign{ \p{\phi}{x} ​&= \d\big(2\,\Omega XD^TD\big) \qquad\qquad \\ }$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.