What is this derivative, with a diagonal matrix? Given matrices $\textbf{A}, \textbf{B}, \textbf{C}$ and column vector $\textbf{v}$, what is the derivative of $\langle \textbf{A} \text{diag}(\textbf{B}\textbf{v}), \textbf{C} \rangle$ with respect to $\textbf{B}$? $\text{diag}(\cdot)$ is a diagonal matrix with the argument as the diagonal and the brackets signify the inner product.
I am trying to find the solution, but am hung up on differentiating the diagonal matrix.
Any assistance would be greatly appreciated. Thank you very much.
 A: The derivative can always be computed by going back to the original limit definition. For a vector-valued function $f: X \rightarrow Y$, its derivative at a point $x$, denoted $Df(x)$, if it exists, is given by
$$[Df(x)]z = \lim_{h \rightarrow 0} h^{-1} (f(x + hz) - f(x)).$$
In other words, if the true (Frechet) derivative exists, then you can compute it by computing directional (Gateaux) derivatives.
In your case, $f(X) = \langle A \mathrm{diag}(Xv), C \rangle$, where $X$ is the variable ($B$ in your original question). Then perhaps you can verify that
$$[Df(X)]Z = \langle A \mathrm{diag}(Zv), C \rangle$$.
A: Let rewrite the cost function
$$
\phi
=
\mathbf{A} \mathrm{diag}(\mathbf{Bv}):
\mathbf{C} 
=
\mathrm{diag}(\mathbf{Bv}):
\mathbf{A}^T \mathbf{C} =
\mathbf{Bv} :
\mathrm{diag}(\mathbf{A}^T \mathbf{C}) = 
\mathrm{diag}(\mathbf{A}^T \mathbf{C})
\mathbf{v^T} : \mathbf{B}
$$
I used the colon operator for the Frobenius inner product.
Also $\mathrm{diag}(\mathbf{X})$ extracts the diagonal element of $\mathbf{X}$ into a vector.
The derivative is
$$
\frac{\partial \phi}{\partial \mathbf{B}}=
\mathrm{diag}(\mathbf{A}^T \mathbf{C})
\mathbf{v^T}
$$
A: When you have a linear map $f : X \to Y$, the derivative at $x$ is $f$. I.e. $d_xf = f$, or in point notation, $(d_xf)h = f(h)$ for all $h$. Your expression, $f(\mathbf{B}) = \langle \mathbf{A}\operatorname{diag}(\mathbf{B}v), \mathbf{C} \rangle$ is linear in $\mathbf{B}$. So the derivative is the linear map
$$d_\mathbf{B}f : \mathbf{H} \mapsto f(\mathbf{H}).$$
Note that this is a linear map from matrices to real numbers, so we cannot represent it as a vector dot product $\nabla f \cdot h$ but we can represent it as a matrix dot product.
Specifically, and unless I am making a mistake somewhere, we have
\begin{align}
(d_\mathbf{B}f) \mathbf{H} &= \langle \mathbf{A}\operatorname{diag}(\mathbf{H}v), \mathbf{C} \rangle \\
&= \operatorname{tr}(\mathbf{C}^\top\mathbf{A}\operatorname{diag}(\mathbf{H}v)) \\
&= \operatorname{tr}((\mathbf{A}^\top\mathbf{C})^\top \operatorname{diag}(\mathbf{H}v)) \\
&= \langle \operatorname{diag}(\mathbf{H}v), \mathbf{A}^\top \mathbf{C} \rangle \\
&= \sum_{i} (\mathbf{H}v)_i (\mathbf{A}^\top \mathbf{C})_{ii} \\
&= \sum_{i} \left(\sum_{j} \mathbf{H}_{ij}v_j \right) (\mathbf{A}^\top \mathbf{C})_{ii} \\
&= \langle \mathbf{H}, (v_j(\mathbf{A}^\top \mathbf{C})_{ii})_{i,j=1}^n \rangle \\
&= \langle \mathbf{H}, \operatorname{Diag}(\mathbf{A}^\top \mathbf{C}) \otimes v \rangle.
\end{align}
Here $\operatorname{Diag}(X) = (X_{ii})_{i = 1}^n$ is the vector of diagonal elements of $X$ and $\otimes$ is the outer product.
So we could say $\operatorname{Diag}(\mathbf{A}^\top \mathbf{C}) \otimes v$ represents the derivative in coordinates.
