Help in calculation of gradient I want to know what is the gradient of tr$(\Lambda^{-1}\Theta^TS\Theta)$ wrt to an element $\Theta_{ij}$ where S and $\Lambda^{-1}$ are symmetric matrices but $\Theta$ is not
 A: First note that if $f(\Theta) = \Lambda^{-1}\Theta^TS\Theta$, then $f(\Theta+ \Delta) = f(\Theta)+\Lambda^{-1}(\Delta^TS\Theta + \Theta^TS\Delta) + \Lambda^{-1}\Delta^TS\Delta$, and we have $\|\Lambda^{-1}\Delta^TS\Delta\| \le K \|\Delta \|^2$, for some $K$.
Hence $Df(\Theta)(\Delta) = \Lambda^{-1}(\Delta^TS\Theta + \Theta^TS\Delta)$.
Let $g(\Theta) = \operatorname{tr} f(\Theta)$.
Since $\operatorname{tr}$ is linear, we have $Dg(\Theta)(\Delta) = \operatorname{tr}( Df(\Theta)(\Delta) ) = \operatorname{tr} ( \Lambda^{-1}(\Delta^TS\Theta + \Theta^TS\Delta ) $. Using the properties of $\operatorname{tr}$, we have:
\begin{eqnarray}
Dg(\Theta)(\Delta) &=& \operatorname{tr} ( \Lambda^{-1}(\Delta^TS\Theta + \Theta^TS\Delta )) \\
&=& \operatorname{tr} ( \Lambda^{-1}\Delta^TS\Theta ) + \operatorname{tr} ( \Lambda^{-1}\Theta^TS\Delta ) \\
&=& \operatorname{tr} ( \Theta^T S \Delta \Lambda^{-1} ) + \operatorname{tr} (   \Lambda^{-1}\Theta^TS\Delta ) \\
&=& \operatorname{tr} ( \Lambda^{-1} \Theta^T S \Delta  ) + \operatorname{tr} (   \Lambda^{-1}\Theta^TS\Delta ) \\
&=& 2 \operatorname{tr} ( \Lambda^{-1} \Theta^T S \Delta  ) \\
&=& 2 \langle S \Theta \Lambda^{-1}, \Delta \rangle
\end{eqnarray}
where the inner product corresponds to the Fronenius norm. Hence we can write
$\nabla g(\theta) = 2S \Theta \Lambda^{-1}$.
