What's the Derivative?

Question

What is the derivative (gradient) of $$\operatorname{vec}(W)^\top \left[\sum_i (ue_i^\top)\otimes(e_ie_i^\top) + \sum_j(e_je_j^\top)\otimes(ue_j^\top) - 2I\otimes I\right] \operatorname{vec}(FF^\top),\tag{3} $$ where $e_i$ denotes the $i$-th vector in the canonical basis and $u=\sum_ie_i=(1,1,\ldots,1)^\top$ where "$\otimes$" denotes the Kronecker product (tensor product) with respect to the rectangular matrix $F$? and $W$ is a constant matrix. The above term can also be simplified as $\sum_{ijl}(F_{il}-F_{jl})^2W_{ij}$ which is a much simpler expression. What would the derivative be? All the above entries are real while $F$ is a rectangular matrix. Also please refer to : Represent in a matrix form: $\sum_{ijl}(F_{il}-F_{jl})^2W_{ij}$ for the details on simplification/algebra.

Answer 1

To find the derivative, it is much easier to work with your original expression $S=\sum_{ijl}(F_{il}-F_{jl})^2W_{ij}$. Note that, for fixed indices $I$ and $J$, \begin{align*} S = &\text{ terms that do not involve } F_{IJ}\\ &+\ (F_{IJ} - F_{IJ})^2 W_{II} + \sum_{j\neq I} (F_{IJ} - F_{jJ})^2 W_{Ij} + \sum_{i\neq I} (F_{iJ} - F_{IJ})^2 W_{iI}\\ = &\text{ terms that do not involve } F_{IJ}\\ &+\ \sum_{k\neq I} (F_{IJ} - F_{kJ})^2 W_{Ik} + \sum_{k\neq I} (F_{kJ} - F_{IJ})^2 W_{kI}. \end{align*} Let $K = W-W^T$. Then \begin{align*} \frac{\partial S}{\partial F_{IJ}} =&2\sum_{k\neq I} (F_{IJ} - F_{kJ}) W_{Ik} + 2\sum_{k\neq I} (F_{kJ} - F_{IJ}) W_{kI}\\ =&2\sum_{k\neq I} (F_{IJ} - F_{kJ}) (W_{Ik}-W_{kI})\\ =&2\sum_{k\neq I} (F_{IJ} - F_{kJ}) K_{Ik}\\ =&2\sum_k (F_{IJ} - F_{kJ}) K_{Ik}\\ =&2F_{IJ}\sum_k K_{Ik} - 2\sum_k K_{Ik}F_{kJ}. \end{align*} Therefore $$ \left(\frac{\partial S}{\partial F_{ij}}\right) =2F\circ(KE) - 2KF $$ where $E$ is the matrix with the same size as $F$ and all entries equal to $1$.