Derivative of the inverse of a symmetric matrix w.r.t itself I'm trying take the derivative of a symmetrix matrix $\mathbf{C}$ with respect to itself.
$$
\begin{equation}
\frac{\partial \mathbf{C}^{-1}}{\partial \mathbf{C}}
\end{equation}
$$
Using the indicial notation, above equation can be rewritten as follows
$$
\begin{equation}
\frac{\partial C_{ij}^{-1}}{\partial C_{kl}}
\end{equation}
$$
At first I've used the following formula,
$$
\begin{equation}
\frac{\partial C_{ij}^{-1}}{\partial C_{kl}} = -C^{-1}_{ik}C^{-1}_{lj}
\end{equation}
$$
But I quickly realized that we've lost the symmetry of the problem now.
I read The Matrix Cookbook and the other posts about the same problem but unfortunately, I couldn't understand the things they've done.
For example in this article, at Eq.(100) authors have used the property below when taking the derivative of Eq.(99)
$$
\begin{equation}
\frac{\partial \mathbf{C}^{-1}}{\partial \mathbf{C}} = -\mathbf{C}^{-1} \boxtimes \mathbf{C}^{-T} \mathbf{I}_s
\end{equation}
$$
Where $\boxtimes$ is the square product, $\mathbf{I}_s$ is the symmetric fourth-order identity tensor and they are defined as follows
$$
\begin{align}
(\mathbf{A} \boxtimes \mathbf{B})_{ijkl} &= \mathbf{A}_{ik}\mathbf{B}_{jl} \\
(\mathbf{I}_s)_{ijkl} &= \frac{1}{2}(\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk})
\end{align}
$$
I couldn't understand how did they achieve this result and how can I derive it myself.
 A: $
\def\p{\partial}\def\o{{\tt1}}
\def\E{{\cal E}}\def\F{{\cal F}}\def\G{{\cal G}}
\def\C{C^{-1}}\def\Ct{C^{-T}}
\def\LR#1{\left(#1\right)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\grad#1#2{\frac{\p #1}{\p #2}}
\def\c#1{\color{red}{#1}}
$Once you learn the technique, the problem can be solved very briefly
$$\eqalign{
d\C &= -{\C\,dC\,\C} \\
 &= -\LR{\C\E\Ct}:dC \\
\grad{\C}{C} &= -{\C\E\Ct} \\
}$$
The details are as follows...

Introduce a fourth-order tensor $\E$ with components
$$\eqalign{
\E_{ijk\ell}
 = \delta_{ik}\,\delta_{j\ell}
 = \begin{cases}
\o\quad{\rm if}\; i=k\;\;{\rm and}\;\;j=\ell \\
0\quad{\rm otherwise}
\end{cases}
\\
}$$
The most useful property of this tensor is its ability to rearrange matrix products
$$\eqalign{
ABC &= \LR{A\E C^T}:B \;=\; \F:B \\
}$$
where juxtaposition implies a single-dot product and a colon $(:)$ denotes the double-dot product
$$\eqalign{
&\F_{ijk\ell}
 = \sum_{p=1}^n\sum_{r=1}^n A_{i\c{p}}\E_{\c{p}jk\c{r}}\,C_{\c{r}\ell}^T 
 \;=\; A_{ik}C_{\ell j} \\
&\LR{\F:B}_{ij}
 = \sum_{k=1}^n\sum_{\ell=1}^n \F_{ij\c{k\ell}}\,B_{\c{k\ell}} 
 = \sum_{k=1}^n\sum_{\ell=1}^n A_{i\c{k}}B_{\c{k\ell}}C_{\c{\ell}j} 
\\
}$$
Start with the differential of the matrix inverse identity
$$\eqalign{
&I = \C C \\
&dI = \c{d\C}C + \C dC \;\doteq\; 0 \\
&\c{d\C} = -\C\,dC\,\C \\
}$$
Then use $\E$ to rearrange the terms and recover the gradient
$$\eqalign{
d\C &= -{\C\E\Ct}:dC \\
\grad{\C}{C} &= -{\C\E\Ct} \\
}$$
Or in component notation
$$\eqalign{
\grad{\C_{ij}}{C_{k\ell}}
 &= -\sum_{p=1}^n\sum_{r=1}^n \C_{i\c{p}}\E_{\c{p}jk\c{r}}\C_{\ell\c{r}}
\;=\; -\C_{ik}\C_{\ell j} \\
}$$
Update
The comments have become a rehash of
the old "symmetric gradient" debate.
On the other hand, if a small set of scalar parameters are used to construct a tensor quantity, then the derivative of the tensor components with respect to one of those scalar parameters can exhibit any number of interesting symmetries.
A large part of Continuum Mechanics is devoted to studying the implications of such symmetries.
But that's a different problem than calculating the derivative of one tensor component with respect to another tensor component. But many people (even professors and famous authors) often conflate these two problems.
A: In the single-variable case, we have that
$$\dfrac{d}{dt}C(t)^{-1}=-C(t)^{-1}\dfrac{dC(t)}{dt}C(t)^{-1}.$$
This can obtained by differentiating the expression $C(t)C(t)^{-1}=I$ on both sides with some simple algebra. This directly generalizes to the multivariable case by expressing
$$C=\sum_{i,j}c_{ij}e_ie_j^T.$$
Then, we have that
$$\dfrac{d}{dc_{kl}}C^{-1}=-C^{-1}\dfrac{dC}{dc_{kl}}C^{-1}=-C^{-1}e_ke_l^TC^{-1},$$
from which we get
$$\dfrac{d}{dc_{kl}}C_{ij}^{-1}=-e_i^TC^{-1}e_ke_l^TC^{-1}e_j=-C_{ik}^{-1}C_{lj}^{-1}.$$
When $C$ is symmetric, then it can be written as
$$C=\sum_{i}c_{ii}e_ie_i^T+\sum_{i>j}c_{ij}(e_ie_j^T+e_je_i^T).$$
$$\begin{array}{rcl}
\dfrac{d}{dc_{kl}}C^{-1}&=&-C^{-1}\dfrac{dC}{dc_{kl}}C^{-1}=-C^{-1}(e_ke_l^T+e_le_k^T)C^{-1},\ \mathrm{for}\ k\ne l\\
\dfrac{d}{dc_{kk}}C^{-1}&=&-C^{-1}\dfrac{dC}{dc_{kk}}C^{-1}=-C^{-1}e_ke_k^TC^{-1}
\end{array}$$
then we have that
$$\begin{array}{rcl}
\dfrac{d}{dc_{kl}}C_{ij}^{-1}&=&-e_i^TC^{-1}(e_ke_l^T+e_le_k^T)C^{-1}e_j=-C_{ik}^{-1}C_{lj}^{-1}-C_{il}^{-1}C_{kj}^{-1},\ \mathrm{for}\ k\ne l\\
\dfrac{d}{dc_{kk}}C_{ij}^{-1}&=&-e_i^TC^{-1}e_ke_k^TC^{-1}e_j=-C_{ik}^{-1}C_{kj}^{-1}.\end{array}$$
