5
$\begingroup$

Let $\mathbf{x}$, $\mathbf{y}$ and $\mathbf{z}$ are $n$-dimensional column vector, and

$$f = \mathbf{x}\circ \mathbf{y} \circ\mathbf{z}$$

Here $\circ$ is the element-wise Hadamard product. Then how to compute the gradient

$\frac{\partial f }{\partial\mathbf{x}}$, $\frac{\partial f }{\partial\mathbf{y}}$, and $\frac{\partial f }{\partial\mathbf{z}}$?

My solution is

$$\frac{\partial f }{\partial\mathbf{x}} = \mathbf{y} \circ\mathbf{z}, \frac{\partial f }{\partial\mathbf{y}} = \mathbf{x} \circ\mathbf{z},\frac{\partial f }{\partial\mathbf{z}} = \mathbf{x} \circ\mathbf{y}$$

Is it correct? thanks.

$\endgroup$
3
  • $\begingroup$ The Hadamard product means you just multiply elements which have the same indices in $\mathbf x,\mathbf y,\mathbf z$. Write for instance the first coordinate, $x_1y_1z_1$. How do you differentiate this wrt $x_1$? How do you differentiate $x_iy_iz_i$ wrt $x_i$? $\endgroup$ Oct 11, 2019 at 23:28
  • $\begingroup$ Cool, is it $\frac{\partial f }{\partial\mathbf{x}} = \mathbf{y} \circ \mathbf{z}$ in vector form? $\endgroup$
    – jason
    Oct 11, 2019 at 23:41
  • 3
    $\begingroup$ Note that $\partial f/\partial \mathbf x$ is a Jacobian, i.e. a matrix. All derivatives vanish except on the diagonal for $x_j$ does not appear in $x_iy_iz_i$ if $i\ne j$. And the diagonal is made of the elements of $\mathbf y\circ\mathbf z$. $\endgroup$ Oct 11, 2019 at 23:45

3 Answers 3

13
$\begingroup$

You can replace the Hadamard product of a vector by a matrix product with the diagonal matrix formed from that vector, e.g. let $$\eqalign{ X &= {\rm Diag}(x), \quad Y &= {\rm Diag}(y), \quad Z &= {\rm Diag}(z) \\ }$$ then the function can be written in a number of ways $$\eqalign{ f &= x\circ y\circ z \\&= YZx = ZXy = XYz \\ }$$ The differential of the function is $$\eqalign{ df &= YZ\,dx + ZX\,dy + XY\,dz \\ }$$ yielding the gradients $$\eqalign{ \frac{\partial f}{\partial x} &= YZ, \quad \frac{\partial f}{\partial y} &= ZX, \quad \frac{\partial f}{\partial z} &= XY \\ }$$ Since diagonal matrices commute, the order of multiplication can be reversed.

$\endgroup$
1
$\begingroup$

If $f\colon \Bbb R^n\times \Bbb R^n \times \Bbb R^n \to \Bbb R^n$ is given by $f({\bf x}, {\bf y}, {\bf z}) = {\bf x}\circ {\bf y} \circ {\bf z}$, then $f$ is trilinear. Thus its total derivative is the linear map ${\rm D}f({\bf x},{\bf y},{\bf z}): \Bbb R^n \times \Bbb R^n \times \Bbb R^n \to \Bbb R^n$ given by $${\rm D}f({\bf x},{\bf y}, {\bf z})({\bf h}_1,{\bf h}_2, {\bf h}_3) = {\bf h}_1 \circ {\bf y} \circ {\bf z} + {\bf x}\circ {\bf h}_2 \circ {\bf z} + {\bf x}\circ {\bf y} \circ {\bf h}_3. $$So we get that $$\frac{\partial f}{\partial {\bf x}}({\bf x},{\bf y},{\bf z}){\bf h} \equiv {\rm D}_{\bf x}f({\bf x}, {\bf y},{\bf z})({\bf h}) = {\bf h}\circ {\bf y} \circ {\bf z},$$ and similarly for the other partial derivatives. Since $\circ$ is commutative, you can write $$\frac{\partial f}{\partial {\bf x}}({\bf x},{\bf y},{\bf z}) = {\bf y}\circ {\bf z}\circ \_.$$

$\endgroup$
1
$\begingroup$

To take the derivative of the Hadamard product, follow equation 30 in the following paper

"Matrix differentiation with diagrammatic notation" https://arxiv.org/pdf/2207.04377.pdf

$\endgroup$
2
  • 2
    $\begingroup$ Would you be willing to summarize these results for us, here on this site? (As part of your new answer) $\endgroup$
    – 311411
    Apr 19, 2023 at 14:46
  • $\begingroup$ Welcome to Math Stack. $\endgroup$
    – 311411
    Apr 19, 2023 at 14:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .