Derivative of Hadamard product What is the derivative of Hadamard product of two matrices with respect to one of them?
I.e. what is $D(AB)$ with respect to $A$?
 A: Consider the 6th-order tensor ${\mathbb M}$, whose components ${\mathbb M}_{ijklmn}$ are unity if $\,(i=k=m)$ and $(j=l=n),\,$ but zero otherwise.
This tensor can be used, along with the Frobenius Product (:), to rewrite the Hadamard Product ($\circ$) between two matrices as
$$\eqalign{
 B\circ A &= B:{\mathbb M}:A \cr\cr
}$$
This makes the finding the gradient (a 4th-order tensor) especially simple
$$\eqalign{
 F &= A\circ B = B\circ A = B:{\mathbb M}:A \cr\cr
dF &= B:{\mathbb M}:dA \cr\cr
\frac{\partial F}{\partial A} &= B:{\mathbb M} \cr
}$$
A: For the derivative with respect to one component of $A$:
$$\frac{\partial}{\partial a_{ij}} (AB)_{kl} = \frac{\partial}{\partial a_{ij}} a_{kl}b_{kl} = b_{kl}\delta_{ik}\delta_{jl}.$$
You'll then have to pick some representation for writing down the full derivative; it is a rank four tensor.
A: Let $f:A\in M_n\rightarrow A\circ B\in M_n$. $f$ is linear ! Then $Df_A:H\in M_n \rightarrow H\circ B$.
EDIT. @ bordeo , the derivative is a linear application; $H$ is any matrix of $M_n$. In particular, the Taylor formula can be written: $f(A+H)=f(A)+Df_A(H)+\cdots$, that is $(A+H)\circ B=A\circ B+H\circ B+\text{"nothing"}$; "nothing" because $f$ is linear and the terms of degree $\geq 2$ in the Taylor formula are $0$.
