Matrix derivation involving element-wise function Let
$$
\mathbf{H}=\sigma(\mathbf{Z})
$$
where $\sigma$ is an element-wise non-linear function, $\mathbf{H}\in\mathbb{R}^{m\times n}$, and $\mathbf{Z}\in\mathbb{R}^{m\times n}$. What is the index notation representation of
$$\frac{\partial \mathbf{H}}{\partial \mathbf{Z}}$$

My attempt:
$$\frac{\partial H_{ij}}{\partial Z_{pq}}=\delta_{ip}\delta_{jq}\sigma'(Z_{ij})$$
 A: $
\def\s{\sigma}\def\t{\s^\prime}
\def\p{\partial}
\def\M{{\cal D}}
\def\LR#1{\left(#1\right)}
\def\op#1{\operatorname{#1}}
\def\trace#1{\op{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\grad#1#2{\frac{\p #1}{\p #2}}
$Given a scalar function $\s(x)$ and its derivative $\t(x)$, we can express the differential of the elementwise application of the function to a matrix argument $Z$ by using an elementwise/Hadamard $(\odot)$ product
$$\eqalign{
H &= \s(Z),\;\; K = \t(Z) \qiq
dH &= K\odot dZ \\
}$$
To convert a Hadamard expression into index notation we need to extend the Kronecker delta from two indices to three, i.e.
$$\eqalign{
{\vec\delta}_{jpr} &= \begin{cases}
1\quad{\rm if}\;\;j=p=r  \\
0\quad{\rm otherwise} \\
\end{cases} \\
}$$
Like the ordinary delta symbol, permuting the indices does not affect the value.
Using one extended delta symbol, we can multiply two vectors (using the Einstein convention)
$$\eqalign{
c &= a\odot b \qiq c_p = a_i\,{\vec\delta}_{ipk}\,b_k \\
}$$
But to multiply matrices we need two such symbols
$$\eqalign{
C
 &= A\odot B \\
C_{pq}
  &= A_{ij}\:{\vec\delta}_{ipk}{\vec\delta}_{jql}\:B_{kl} 
 \;=\; A_{ij}\:\M_{ijpqkl}\:B_{kl} \\
\M_{ijpqkl} &= \begin{cases}
1 \quad {\rm if}\;i=p=k\;\;{\rm and}\;\;j=q=l \\
0 \quad {\rm otherwise} \\
\end{cases}
\\
}$$
Applying this to the differential expression above yields
$$\eqalign{
dH_{pq} &= K_{ij}\,\M_{ijpqkl}\:dZ_{kl}
 \qiq \grad{H_{pq}}{Z_{kl}} = K_{ij}\,\M_{ijpqkl} \\
}$$
If you suspend the summation convention, then the result can be written as you have done in your post.
