2
$\begingroup$

There is a scaler-by-matrix derivative identity:

$$\frac{\partial}{\partial X}trace\left(AXBX'C\right)=B'X'A'C'+BX'CA$$

How does this change if instead I am trying to find

$$\frac{\partial}{\partial x}trace\left(Adiag(x)Bdiag(x)'C\right)$$

where $x$ is a vector rather than a matrix.

My thinking is that all I have to do is multiply the original identity by a vector of ones as that would be the derivative of $diag(x)$. However, I'm not sure how the chain rule interacts with traces.

I ask as I am trying to calculate. $$\frac{\partial}{\partial w}trace\left(Ddiag(w)\Omega diag(w)D'\right)$$

where $w \mathbb{\in R^{N}}$, $D\mathbb{\in R^{M\times N}}$, and $\Omega\mathbb{\in R^{N\times N}}$. Also $\Omega$ can be assumed to be positive definite.

This implies the result would be

$$\left(2\Omega diag(w)D'D\right)e$$

where $e \mathbb{\in R^{N}}$ is a vector of ones.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Don't multiply the matrix by $e$, but create a vector from its diagonal elements $${\rm diag}\Big(2\Omega\,\,{\rm Diag}(w)\,D'D\Big)$$ $\endgroup$ – greg Jan 19 '18 at 3:44
  • $\begingroup$ @greg I appreciate this. Last night I had manually calculated the derivative when $w$ was length 5. When I got around to putting it in the computer today, I was able to verify that it was equivalent to your result. I would be happy to accept yours as the answer if you put it as one. $\endgroup$ – John Jan 19 '18 at 16:21
2
$\begingroup$

Let $f : \mathbb R^n \to \mathbb R$ be defined by

$$f (\mathrm x) := \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \right)$$

where $\mathrm A \in \mathbb R^{m \times n}$, $\mathrm B \in \mathbb R^{n \times n}$ and $\mathrm C \in \mathbb R^{n \times m}$ are given. The directional derivative of $f$ in the direction of $\mathrm v \in \mathbb R^n$ at $\mathrm x \in \mathbb R^n$ is given by

$$\begin{array}{rl} \displaystyle\lim_{h \to 0} \dfrac{f (\mathrm x + h \,\mathrm v) - f (\mathrm x)}{h} &= \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm v) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \right) + \mbox{tr} \left( \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \, \mbox{diag} (\mathrm v) \, \mathrm C \right)\\ &= \mbox{tr} \left( \mbox{diag} (\mathrm v) \, \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mbox{tr} \left( \mbox{diag} (\mathrm v) \, \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)\\ &= \mathrm v^\top \mbox{diag}^{-1} \left( \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mathrm v^\top \mbox{diag}^{-1} \left( \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)\end{array}$$

where $\mbox{diag}^{-1} : \mathbb R^{n \times n} \to \mathbb R^n$ is a linear function that takes a square matrix and extracts its main diagonal as a column vector. Thus, the gradient of $f$ is

$$\nabla_{\mathrm x} f(\mathrm x) = \color{blue}{\mbox{diag}^{-1} \left( \mathrm B \, \mbox{diag} (\mathrm x) \, \mathrm C \, \mathrm A \right) + \mbox{diag}^{-1} \left( \mathrm C \, \mathrm A \, \mbox{diag} (\mathrm x) \, \mathrm B \right)}$$

|cite|improve this answer
$\endgroup$
  • $\begingroup$ BTW, I really like the notation for inverse diagonal that you used. $\endgroup$ – John Jan 22 '18 at 17:40
  • 1
    $\begingroup$ I didn't invent it. Take a look at page 10 of this. $\endgroup$ – Rodrigo de Azevedo Jan 22 '18 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.