Differentiating with Hadamard: $Ax \odot b$ I'm trying to differentiate $\mathbf{A}\mathbf{x}\odot \mathbf{b}$, with respect to $\mathbf{x}$, where $\odot$ is the Hadamard/entrywise product.
I tried making a simple example where $\mathbf{A}\in \mathbf{R}^{2\times2}$, and multiplying out:
$$\left[ {\begin{array}{cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array}} \right] \left[ {\begin{array}{c} x_1 \\ x_2 \end{array}} \right] \odot \left[ {\begin{array}{c} b_1 \\ b_2 \end{array}} \right] = \left[ {\begin{array}{cc} b_1(a_{11}x_1+a_{12}x_2) \\ b_2(a_{21}x_1+a_{22}x_2) \end{array}} \right]$$
Then, $\frac{d}{d\mathbf{x}}$ is $$\left[ {\begin{array}{cc} b_1a_{11} & b_1a_{12} \\ b_2a_{21} & b_2a_{22} \end{array}} \right]$$
In matrix notation that looks like $\mathbf{A}$ entry wise multiplied by some "double-column" $\mathbf{b}$, but I can't see what this would be. 
Is this correct? And how would you write this in matrix notation?
 A: The Hadamard product between vectors can be written in a number of ways 
$$\eqalign{
v\circ b &= b\circ v  &= Vb &= Bv \cr
}$$
where $\,\,B={\rm Diag}(b)$ and $\,V={\rm Diag}(v)$.

Which allows you to write the function in a form which is easier to work with 
$$\eqalign{
 y &= b\circ(Ax) \cr
   &= B\,(Ax) \cr
   &= BA\,x \cr\cr
dy &= BA\,dx \cr\cr
\frac{\partial y}{\partial x} &= BA \cr
 &= {\rm Diag}(b)A \cr\cr
}$$
A: In index notation,
$$
(\mathbf{A}\mathbf{x}\odot \mathbf{b})_i=(\mathbf{A}\mathbf{x})_i b_i=\sum_{j}A_{ij}x_jb_i,
$$
so
$$
\frac{\partial}{\partial x_{j}}(\mathbf{A}\mathbf{x}\odot \mathbf{b})_i=b_i A_{ij};
$$
i.e., the derivative matrix is ${\mathbf{A}}$, with each row multiplied by the corresponding element of $\mathbf{b}$.  As pointed out in the other answer, this is equivalent to $$\nabla_x (\mathbf{b} \odot \mathbf{A}\mathbf{x}) = \text{diag}(\mathbf{b}) \mathbf{A}.$$
A: The function $f\colon \Bbb R^n \to \Bbb R^n$ given by $f({\bf x}) = A{\bf x}\odot {\bf b}$ is linear, so it's total derivative is itself. Namely, for all ${\bf x} \in \Bbb R^n$, we have that $Df({\bf x})\colon \Bbb R^n \to \Bbb R^n$ is given by $$Df({\bf x})({\bf h}) = f({\bf h}) = A{\bf h}\odot {\bf b}.$$Then the $i$-th entry of $$ Df({\bf x})({\bf e}_j)=\left(\sum_{k=1}^n a_{kj}{\bf e}_k\right)\odot {\bf b} = \sum_{k=1}^n a_{kj} {\bf e}_k \odot {\bf b} $$is just $$\sum_{k=1}^na_{kj}\delta_{ki}b_i = a_{ij}b_i.$$This means that the matrix representing $Df({\bf x})$ relative to the standard basis of $\Bbb R^n$ is ${\rm diag}({\bf b})A$.
