Derivative of euclidean norm of matrix-vector product Intuitively, I think this should follow the chain rule and be the following: 
$$ \frac{d}{d x} \left \| A\vec{x} \right \|_{2} = \frac{(A\vec{x})^{T}}{\left \| A\vec{x} \right \|_{2}} \cdot A$$
Is this correct or am I overlooking something?
 A: I agree. Your application of the chain-rule was correct.
$$\frac{d\Big(||Ax||_2\Big)}{dx} = \frac{d\Big(||Ax||_2\Big)}{d(Ax)}\frac{d(Ax)}{dx} = \frac{(Ax)^T}{||Ax||_2}A$$
I don't want to just say "yes" as an answer though, so below I have worked out the solution explicitly for the $\mathbb{R}^2$ case just to make you more comfortable that this is correct. Let $a_{ij}$ be the components of $A$ and let $x_j$ be the components of $x$. We have,
$$\frac{d}{dx_1}||Ax||_2 = \frac{d}{dx_1}\Big(\sqrt{(a_{11}x_1 + a_{12}x_2)^2 + (a_{21}x_1 + a_{22}x_2)^2}\Big)$$
$$=\frac{1}{2}\frac{2(a_{11}x_1 + a_{12}x_2)a_{11}+2(a_{21}x_1 + a_{22}x_2)a_{21}}{\sqrt{(a_{11}x_1 + a_{12}x_2)^2 + (a_{21}x_1 + a_{22}x_2)^2}}$$
$$=\frac{(a_{11}x_1 + a_{12}x_2)a_{11}+(a_{21}x_1 + a_{22}x_2)a_{21}}{\sqrt{(a_{11}x_1 + a_{12}x_2)^2 + (a_{21}x_1 + a_{22}x_2)^2}}$$
$$=\frac{(Ax)^T\begin{bmatrix}a_{11}\\a_{21}\end{bmatrix}}{||Ax||_2}$$
and similarly for $\frac{d}{dx_2}$ we get,
$$\frac{d}{dx_2}||Ax||_2=\frac{(Ax)^T\begin{bmatrix}a_{12}\\a_{22}\end{bmatrix}}{||Ax||_2}$$
So by inspection,
$$\frac{d}{dx}||Ax||_2 = \begin{bmatrix}\frac{d}{dx_1}||Ax||_2 & \frac{d}{dx_2}||Ax||_2\end{bmatrix} = \frac{(Ax)^TA}{||Ax||_2} = \frac{x^TA^TA}{||Ax||_2}$$
