# Derivative of Hadamard product with kernels

I want to calculate $$\frac{\partial(A \circ X^\top X)}{\partial(X)},$$ where $\circ$ is Hadamard product (elementwise product), $X \in R^{r \times n}$, $A \in R^{n \times n}$, and $\frac{\partial A}{\partial X}=0$.

So far, I found that $$\frac{\partial(A \circ B)}{\partial(C)} = \frac{\partial(A)}{\partial(C)} \circ B + A \circ \frac{\partial(B)}{\partial(C)},$$ from here (generic rule matrix differentiation (Hadamard Product, element-wise))

In my case, $B=X^\top X$ and $C=X$.

Therefore, $$\frac{\partial(A \circ X^\top X)}{\partial(X)} = A \circ 2X^\top .$$

However, $A\in R^{n\times n}$ and $X^\top \in R^{n \times r}$.

Therefore, I can't do the Hadamard product.

How can I do this?

First we'll need a few tensors.

A 6th-order tensor ${\mathbb M}$, whose components ${\mathbb M}_{ijklmn}$ are unity if $\,(i=k=m)$ and $(j=l=n),\,$ but zero otherwise.

This tensor makes it possible to rewrite a Hadamard ($\circ$) product using Frobenius (:) products like so \eqalign{ A\circ Z &= A:{\mathbb M}:Z \cr\cr }

Next we'll need two 4th-order isotropic tensors whose components are \eqalign{ {\mathbb E}_{ijkl} &= \delta_{ik}\,\delta_{jl} \cr {\mathbb B}_{ijkl} &= \delta_{il}\,\delta_{jk} \cr } These tensors make it possible to re-arrange matrix products \eqalign{ A\,dX\,Z &= A{\mathbb E}Z^T:dX \cr A\,dX^T\,Z &= A{\mathbb E}Z^T:{\mathbb B}:dX \cr\cr }

Now we are ready to find the differential and gradient of your function \eqalign{ F &= A\circ X^TX \cr &= A:{\mathbb M}:X^TX \cr\cr dF &= A:{\mathbb M}:(dX^TX+X^TdX) \cr &= A:{\mathbb M}:({\mathbb E}X^T:{\mathbb B}:dX+X^T{\mathbb E}:dX) \cr &= A:{\mathbb M}:{\mathbb E}X^T:{\mathbb B}:dX + A:{\mathbb M}:X^T{\mathbb E}:dX \cr &= A:{\mathbb M}:\big({\mathbb E}X^T:{\mathbb B}\,\,+\,\,X^T{\mathbb E}\big):dX \cr\cr \frac{\partial F}{\partial X} &= A:{\mathbb M}:\Big({\mathbb E}X^T:{\mathbb B} \,+\, X^T{\mathbb E}\Big) \cr\cr\cr } Another approach is to use vectorization.
Let \eqalign{ f &= \operatorname{vec}(F) \cr x &= \operatorname{vec}(X) \cr a &= \operatorname{vec}(A) \cr {\mathcal A} &= \operatorname{Diag}(a) \cr } Then \eqalign{ df &= a\circ\operatorname{vec}(dX^TX+X^TdX) \cr &= {\mathcal A}\,\Big((X^T\otimes I)B\,dx + (I\otimes X^T)\,dx \Big) \cr\cr \frac{\partial f}{\partial x} &= {\mathcal A}\,(X^T\otimes I)B\,+\,{\mathcal A}\,(I\otimes X^T) \cr }where $B$ is the Kronecker Commutation matrix.

This is actually quite similar to the tensor result, with \eqalign{ {\mathcal A} &\sim A:{\mathbb M} \cr (X^T\otimes I)B &\sim I\,{\mathbb E}\,X^T:{\mathbb B} \cr (I\otimes X^T) &\sim X^T{\mathbb E}\,I \cr }

• can you recommend some material to study what you show to us? @greg – user1498253 Sep 4 '17 at 11:24