# What's the Derivative?

What is the derivative (gradient) of $$\operatorname{vec}(W)^\top \left[\sum_i (ue_i^\top)\otimes(e_ie_i^\top) + \sum_j(e_je_j^\top)\otimes(ue_j^\top) - 2I\otimes I\right] \operatorname{vec}(FF^\top),\tag{3}$$ where $e_i$ denotes the $i$-th vector in the canonical basis and $u=\sum_ie_i=(1,1,\ldots,1)^\top$ where "$\otimes$" denotes the Kronecker product (tensor product) with respect to the rectangular matrix $F$? and $W$ is a constant matrix. The above term can also be simplified as $\sum_{ijl}(F_{il}-F_{jl})^2W_{ij}$ which is a much simpler expression. What would the derivative be? All the above entries are real while $F$ is a rectangular matrix. Also please refer to : Represent in a matrix form: $\sum_{ijl}(F_{il}-F_{jl})^2W_{ij}$ for the details on simplification/algebra.

• The whole left side of the expression is independent of $F$, so all you need to do is calculate the vector derivative of $\mathrm{vec}(FF^T)$. May 28, 2013 at 7:17
• Considering the expression on the left to be $C$, the derivative of $CFF^T$ would be $(C+C^T)F$, I guess? Now with $C\operatorname{vec}(FF^T)$, am confused.. May 28, 2013 at 16:15

To find the derivative, it is much easier to work with your original expression $S=\sum_{ijl}(F_{il}-F_{jl})^2W_{ij}$. Note that, for fixed indices $I$ and $J$, \begin{align*} S = &\text{ terms that do not involve } F_{IJ}\\ &+\ (F_{IJ} - F_{IJ})^2 W_{II} + \sum_{j\neq I} (F_{IJ} - F_{jJ})^2 W_{Ij} + \sum_{i\neq I} (F_{iJ} - F_{IJ})^2 W_{iI}\\ = &\text{ terms that do not involve } F_{IJ}\\ &+\ \sum_{k\neq I} (F_{IJ} - F_{kJ})^2 W_{Ik} + \sum_{k\neq I} (F_{kJ} - F_{IJ})^2 W_{kI}. \end{align*} Let $K = W-W^T$. Then \begin{align*} \frac{\partial S}{\partial F_{IJ}} =&2\sum_{k\neq I} (F_{IJ} - F_{kJ}) W_{Ik} + 2\sum_{k\neq I} (F_{kJ} - F_{IJ}) W_{kI}\\ =&2\sum_{k\neq I} (F_{IJ} - F_{kJ}) (W_{Ik}-W_{kI})\\ =&2\sum_{k\neq I} (F_{IJ} - F_{kJ}) K_{Ik}\\ =&2\sum_k (F_{IJ} - F_{kJ}) K_{Ik}\\ =&2F_{IJ}\sum_k K_{Ik} - 2\sum_k K_{Ik}F_{kJ}. \end{align*} Therefore $$\left(\frac{\partial S}{\partial F_{ij}}\right) =2F\circ(KE) - 2KF$$ where $E$ is the matrix with the same size as $F$ and all entries equal to $1$.
• also that's the Hadamard Product between $F$ and $KE$. Right? Jun 13, 2013 at 20:39
• There is a sign error in the second term of the derivative. The derivative of $(F_{kJ}-F_{IJ})^2$ with respect to $F_{IJ}$ is $-2\,(F_{kJ}-F_{IJ})$. However, the form of the final result is correct, if you redefine $K=W+W^T$.