# Taking the matrix derivative of the product of one matrix and a Hadamard Product.

Consider the three matrices $$\mathbf{C}$$, $$\mathbf{A}$$, and $$\mathbf{T}$$. The matrix $$\mathbf{C}$$ has $$\mathit{m} \times \mathit{k}$$ entries, $$\mathbf{A}$$ is a $$\mathit{k} \times \mathit{n}$$ matrix, and $$\mathbf{T}$$ is a $$\mathit{m} \times \mathit{n}$$ matrix.

I'd like to evaluate the following matrix derivative: $$\frac{\partial}{\partial\mathbf{C}}\bigl( (\mathbf{C}\mathbf{A}) \circ \mathbf{T} \bigr)$$

Where $$\circ$$ represents the Hadamard product. Note that the dimensions of this expression are consistent since $$\mathbf{CA}$$ is a $$\mathit{m} \times \mathit{n}$$ matrix. Note that both $$\mathbf{A}$$ and $$\mathbf{T}$$ are both constant matrices with respect to $$\mathbf{C}$$.

I'm wondering how I can evaluate and then express this result. I know that since the expression I am taking the derivative of is a $$\mathit{m}$$ $$\times$$ $$\mathit{n}$$ matrix, and $$\mathbf{C}$$ is a $$\mathit{m} \times \mathit{k}$$ matrix, that the result of this derivative expression will have $$\mathit{m} \times \mathit{n} \times \mathit{m} \times \mathit{k}$$ entries.

I'd appreciate any answer, including one in index notation.

• Probably the clearest way to express the derivative is to write out the matrix $\frac{\partial }{\partial C_{ij}}((CA) \circ T)$. Commented Jul 20, 2020 at 10:18
Because $$C \mapsto (CA) \circ T$$ is a linear map, it is very easy to compute the derivative in differential form. In particular, we have $$D_C(C_0)(dC) = (dC\,A) \circ T.$$ Now, let $$E_{ij}$$ denote the matrix with a $$1$$ in the $$i,j$$ entry and zeros elsewhere. Let $$e_i$$ denote the column vector with a $$1$$ in the $$i$$ entry and zeros elsewhere. We have $$\frac{\partial f}{\partial C_{ij}}|_{C = C_0} = D_C(C_0)(E_{ij}) = (E_{ij}A) \circ T = (e_ie_j^\top A \circ T) = e_i(A^\top e_j)^\top \circ T\\ = E_{ii} T \operatorname{diag}(A^\top e_j).$$ In index notation, one might write $$\frac{\partial f_{pq}}{\partial C_{ij}} = e_p^TE_{ii} T \operatorname{diag}(A^\top e_j)e_q = \delta_{ip} e_i^\top T A_{jq}e_q = \delta_{ip} A_{jq}T_{iq},$$ with no summation implied.
It's probably simpler to vectorize the matrix equation, and then to eliminate the Hadamard product in favor of multiplication by a diagonal matrix, i.e. \eqalign{ F &= T\circ CA \\ {\rm vec}(F) &= {\rm vec}(T)\circ {\rm vec}(CA) \\ &= {\rm Diag}\big({\rm vec}(T)\big)\,(A^T\otimes I)\,{\rm vec}(C) \\ \frac{\partial f}{\partial c} = \frac{\partial\,{\rm vec}(F)}{\partial\,{\rm vec}(C)} &= {\rm Diag}\big({\rm vec}(T)\big)\,(A^T\otimes I) \\ } If you really want a fourth order tensor, there is a straightforward one-to-one mapping between the matrix and its vectorized form, e.g. \eqalign{ F &\in {\mathbb R}^{m\times n} \quad\iff\quad f \in {\mathbb R}^{mn\times 1} \\ F_{ij} &= f_{\alpha} \\ \alpha &= i+(j-1)\,m \\ i &= 1+(\alpha-1)\,{\rm mod}\,m \\ j &= 1+(\alpha-1)\,{\rm div}\,m \\ } So you can convert the gradient matrix into a tensor
\eqalign{ \frac{\partial f_\alpha}{\partial c_\beta} = \frac{\partial F_{ij}}{\partial C_{k\ell}} \\ }