# How to compute derivative with Hadamard product?

Let $$\mathbf{x}$$, $$\mathbf{y}$$ and $$\mathbf{z}$$ are $$n$$-dimensional column vector, and

$$f = \mathbf{x}\circ \mathbf{y} \circ\mathbf{z}$$

Here $$\circ$$ is the element-wise Hadamard product. Then how to compute the gradient

$$\frac{\partial f }{\partial\mathbf{x}}$$, $$\frac{\partial f }{\partial\mathbf{y}}$$, and $$\frac{\partial f }{\partial\mathbf{z}}$$?

My solution is

$$\frac{\partial f }{\partial\mathbf{x}} = \mathbf{y} \circ\mathbf{z}, \frac{\partial f }{\partial\mathbf{y}} = \mathbf{x} \circ\mathbf{z},\frac{\partial f }{\partial\mathbf{z}} = \mathbf{x} \circ\mathbf{y}$$

Is it correct? thanks.

• The Hadamard product means you just multiply elements which have the same indices in $\mathbf x,\mathbf y,\mathbf z$. Write for instance the first coordinate, $x_1y_1z_1$. How do you differentiate this wrt $x_1$? How do you differentiate $x_iy_iz_i$ wrt $x_i$? Oct 11, 2019 at 23:28
• Cool, is it $\frac{\partial f }{\partial\mathbf{x}} = \mathbf{y} \circ \mathbf{z}$ in vector form? Oct 11, 2019 at 23:41
• Note that $\partial f/\partial \mathbf x$ is a Jacobian, i.e. a matrix. All derivatives vanish except on the diagonal for $x_j$ does not appear in $x_iy_iz_i$ if $i\ne j$. And the diagonal is made of the elements of $\mathbf y\circ\mathbf z$. Oct 11, 2019 at 23:45

## 3 Answers

You can replace the Hadamard product of a vector by a matrix product with the diagonal matrix formed from that vector, e.g. let \eqalign{ X &= {\rm Diag}(x), \quad Y &= {\rm Diag}(y), \quad Z &= {\rm Diag}(z) \\ } then the function can be written in a number of ways \eqalign{ f &= x\circ y\circ z \\&= YZx = ZXy = XYz \\ } The differential of the function is \eqalign{ df &= YZ\,dx + ZX\,dy + XY\,dz \\ } yielding the gradients \eqalign{ \frac{\partial f}{\partial x} &= YZ, \quad \frac{\partial f}{\partial y} &= ZX, \quad \frac{\partial f}{\partial z} &= XY \\ } Since diagonal matrices commute, the order of multiplication can be reversed.

If $$f\colon \Bbb R^n\times \Bbb R^n \times \Bbb R^n \to \Bbb R^n$$ is given by $$f({\bf x}, {\bf y}, {\bf z}) = {\bf x}\circ {\bf y} \circ {\bf z}$$, then $$f$$ is trilinear. Thus its total derivative is the linear map $${\rm D}f({\bf x},{\bf y},{\bf z}): \Bbb R^n \times \Bbb R^n \times \Bbb R^n \to \Bbb R^n$$ given by $${\rm D}f({\bf x},{\bf y}, {\bf z})({\bf h}_1,{\bf h}_2, {\bf h}_3) = {\bf h}_1 \circ {\bf y} \circ {\bf z} + {\bf x}\circ {\bf h}_2 \circ {\bf z} + {\bf x}\circ {\bf y} \circ {\bf h}_3.$$So we get that $$\frac{\partial f}{\partial {\bf x}}({\bf x},{\bf y},{\bf z}){\bf h} \equiv {\rm D}_{\bf x}f({\bf x}, {\bf y},{\bf z})({\bf h}) = {\bf h}\circ {\bf y} \circ {\bf z},$$ and similarly for the other partial derivatives. Since $$\circ$$ is commutative, you can write $$\frac{\partial f}{\partial {\bf x}}({\bf x},{\bf y},{\bf z}) = {\bf y}\circ {\bf z}\circ \_.$$

To take the derivative of the Hadamard product, follow equation 30 in the following paper

"Matrix differentiation with diagrammatic notation" https://arxiv.org/pdf/2207.04377.pdf

• Would you be willing to summarize these results for us, here on this site? (As part of your new answer) Apr 19, 2023 at 14:46
• Welcome to Math Stack. Apr 19, 2023 at 14:46