The gradient of an element wise matrix function Let $F = F[A]$ represent an element wise function $F$ applied to matrix $A$. Here $F_{ij} = f(A_{ij})$ where $f$ is a scalar function. I would like to derive an expression for $\frac{\partial F}{\partial A}$. 
My strategy was to use summation notation:
$$\frac{\partial F_{ij}}{\partial A_{pq}} = \frac{\partial f}{\partial A_{ij}} \frac{\partial A_{ij}}{\partial A_{pq}} $$
$$\frac{\partial F_{ij}}{\partial A_{pq}} = \frac{\partial f}{\partial A_{ij}} \delta_{ip} \delta_{jq}$$
I know there should be 4th order tensor result but the implied sum is throwing me off. I am not too familiar with matrix manipulation when there are tensors of order 3 and higher so I did not try to construct a differential.
Any walkthroughs/strategies would be much appreciated!
 A: Given a function $f=f(\alpha)$ of a scalar argument $\alpha$, its derivative $f'(\alpha)=\frac{df}{d\alpha}$ can be used to write the differential as 
$$\eqalign{
df &= f'\,d\alpha \\
}$$
Applying these functions element-wise to a matrix argument $A$ defines the matrices
$$\eqalign{
F  &= f(A) \\
F' &= f'(A) \\
}$$
Write matrix differential using the element-wise product $\odot$ (aka the Hadamard product). 
$$\eqalign{
dF &= F'\odot dA \\
}$$
Introduce a sixth-order tensor ${\cal S}$ with components
$$\eqalign{
{\cal S}_{ijklmn} &= \begin{cases}
1\quad{\rm if}\;(i=k=m)\;{\rm and}\;(j=l=n) \\
0\quad{\rm otherwise}
\end{cases}  \\
}$$
Note that this tensor is unchanged under permutation of its the three index-pairs, so
$${\cal S}_{ij\,kl\,mn} = {\cal S}_{kl\,mn\,ij} = {\cal S}_{ij\,mn\,kl}$$
The Hadamard product between two matrices can now be written in index notation.
$$\eqalign{
C &= A\odot B \quad&\implies\quad &C_{kl} = A_{ij}{\cal S}_{ijklmn}B_{mn} \\
  &= A:{\cal S}:B \quad&&\big({\rm Double\,Dot\,Products}\big) \\
}$$
Apply this to the differential relationship and solve for the gradient $\Gamma$ (a fourth-order tensor).
$$\eqalign{
dF &= F':{\cal S}:dA \\
\frac{\partial F}{\partial A}
 &= F':{\cal S}
 \;\triangleq\; \Gamma \\
 &= {\cal S}:F' \quad\big({\rm Permute\,Indices\,of\,}{\cal S}\big) \\
\frac{\partial F_{ij}}{\partial A_{kl}} 
 &= {\cal S}_{ijklmn}\,F'_{mn} 
 \;=\; \Gamma_{ijkl} \\
\\
}$$
NB: This answer uses a convention wherein repeated indices are (implicitly) summed over.
