I want to calculate $$ \frac{\partial(A \circ X^\top X)}{\partial(X)}, $$ where $ \circ $ is Hadamard product (elementwise product), $X \in R^{r \times n}$, $A \in R^{n \times n}$, and $\frac{\partial A}{\partial X}=0$.

So far, I found that $$ \frac{\partial(A \circ B)}{\partial(C)} = \frac{\partial(A)}{\partial(C)} \circ B + A \circ \frac{\partial(B)}{\partial(C)},$$ from here (generic rule matrix differentiation (Hadamard Product, element-wise))

In my case, $B=X^\top X$ and $C=X$.

Therefore, $$ \frac{\partial(A \circ X^\top X)}{\partial(X)} = A \circ 2X^\top .$$

However, $A\in R^{n\times n}$ and $X^\top \in R^{n \times r}$.

Therefore, I can't do the Hadamard product.

How can I do this?


1 Answer 1


First we'll need a few tensors.

A 6th-order tensor ${\mathbb M}$, whose components ${\mathbb M}_{ijklmn}$ are unity if $\,(i=k=m)$ and $(j=l=n),\,$ but zero otherwise.

This tensor makes it possible to rewrite a Hadamard ($\circ$) product using Frobenius (:) products like so $$\eqalign{ A\circ Z &= A:{\mathbb M}:Z \cr\cr }$$

Next we'll need two 4th-order isotropic tensors whose components are $$\eqalign{ {\mathbb E}_{ijkl} &= \delta_{ik}\,\delta_{jl} \cr {\mathbb B}_{ijkl} &= \delta_{il}\,\delta_{jk} \cr }$$ These tensors make it possible to re-arrange matrix products $$\eqalign{ A\,dX\,Z &= A{\mathbb E}Z^T:dX \cr A\,dX^T\,Z &= A{\mathbb E}Z^T:{\mathbb B}:dX \cr\cr }$$

Now we are ready to find the differential and gradient of your function $$\eqalign{ F &= A\circ X^TX \cr &= A:{\mathbb M}:X^TX \cr\cr dF &= A:{\mathbb M}:(dX^TX+X^TdX) \cr &= A:{\mathbb M}:({\mathbb E}X^T:{\mathbb B}:dX+X^T{\mathbb E}:dX) \cr &= A:{\mathbb M}:{\mathbb E}X^T:{\mathbb B}:dX + A:{\mathbb M}:X^T{\mathbb E}:dX \cr &= A:{\mathbb M}:\big({\mathbb E}X^T:{\mathbb B}\,\,+\,\,X^T{\mathbb E}\big):dX \cr\cr \frac{\partial F}{\partial X} &= A:{\mathbb M}:\Big({\mathbb E}X^T:{\mathbb B} \,+\, X^T{\mathbb E}\Big) \cr\cr\cr }$$ Another approach is to use vectorization.
Let $$\eqalign{ f &= \operatorname{vec}(F) \cr x &= \operatorname{vec}(X) \cr a &= \operatorname{vec}(A) \cr {\mathcal A} &= \operatorname{Diag}(a) \cr }$$ Then $$\eqalign{ df &= a\circ\operatorname{vec}(dX^TX+X^TdX) \cr &= {\mathcal A}\,\Big((X^T\otimes I)B\,dx + (I\otimes X^T)\,dx \Big) \cr\cr \frac{\partial f}{\partial x} &= {\mathcal A}\,(X^T\otimes I)B\,+\,{\mathcal A}\,(I\otimes X^T) \cr }$$where $B$ is the Kronecker Commutation matrix.

This is actually quite similar to the tensor result, with $$\eqalign{ {\mathcal A} &\sim A:{\mathbb M} \cr (X^T\otimes I)B &\sim I\,{\mathbb E}\,X^T:{\mathbb B} \cr (I\otimes X^T) &\sim X^T{\mathbb E}\,I \cr }$$

  • 2
    $\begingroup$ can you recommend some material to study what you show to us? @greg $\endgroup$ Commented Sep 4, 2017 at 11:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .