Derivative of $\|Xa\|_2 $ with respect to $X$ Can someone give me the answer to the following expression?
$\frac{\partial}{\partial X}\|Xa\|_2 =?$
$X$ is a square matrix and $a$ is a vektor of the apropriate size. $\|\cdot\|_2$ is the euclidean norm.
Thanks
 A: Take any matrix $B$ and expand 
$$
\|(X+tB)a\|_2^2 = \|Xa\|_2^2 + 2 t \langle Xa , Ba \rangle + t^2 \|Ba\|_2^2.
$$
Therefore the directional derivative of the square of your map in the direction $B$ is
$$
2 B^t Xa
$$
where $B^t$ is the transposed matrix of $B$.
A: Recall that, when it exists, the differential $DF(X)$ at $X$ of a real-valued function $F$ defined on the space $M_n$ of the square matrices of size $n$ is the unique linear functional $L_X$ defined on $M_n$ such that $F(X+Y)=F(X)+L_X(Y)+\|Y\|_2\cdot\varepsilon_X(Y)$, where $\varepsilon_X(Y)$ has limit $0$ when $Y$ converges to the zero matrix.
Let $G:X\mapsto\|Xa\|_2^2$. Then $G(X)=a^tX^tXa$ and $
G(X+Y)=G(X)+2a^tX^tYa+a^tY^tYa,
$
hence $DG(X)$ exists and is given by
$$
DG(X)(Y)=2a^tX^tYa. 
$$
Let $F:X\mapsto\|Xa\|_2=\sqrt{G(X)}$. Assume first that $Xa\ne0$. Then,
$$
G(X+Y)=G(X)\cdot(1+F(X)^{-2}\cdot DG(X)(Y)+o(\|Y\|_2)).
$$ 
Together with the fact that $\sqrt{1+t}=1+\frac12t+o(t)$ when $t\to0$, this yields
$$
F(X+Y)=F(X)\cdot(1+\tfrac12F(X)^{-2}\cdot DG(X)(Y)+o(\|Y\|_2)),
$$
hence
$$
F(X+Y)=F(X)+\tfrac12F(X)^{-1}\cdot DG(X)(Y)+o(\|Y\|_2).
$$
Thus, $DF(X)$ exists and is given by
$$
DF(X)(Y)=\tfrac12F(X)^{-1}\cdot DG(X)(Y)=\frac{a^tX^tYa}{\|Xa\|_2}.
$$
The case $Xa=0$ is different. Now,
$F(X+Y)=F(X)+\|Ya\|_2$. 


*

*If $a=0$, $F$ is differentiable at $X$ and $DF(X)=0$ (in this case, $F=0$ everywhere).

*If $a\ne0$, $F$ is not differentiable at $X$ just like the function $t\mapsto|t|$ is not differentiable at $t=0$ (for example the function $Y\mapsto\|Ya\|_2$ is even, hence it cannot be linear at first order unless being zero).

A: Given the function
\begin{eqnarray*}
f\left(X\right) & = & \left\Vert X\cdot a\right\Vert \\
 & = & \sqrt{a^{T}\cdot X^{T}\cdot X\cdot a}\\
 & = & \left(a^{T}\cdot X^{T}\cdot X\cdot a\right)^{1/2}
\end{eqnarray*}
where $X$ is a matrix and $a$ is a vector. The gradient is
\begin{eqnarray*}
\frac{\partial f}{\partial X} & = & \frac{1}{2}\cdot\left(a^{T}\cdot X^{T}\cdot X\cdot a\right)^{-1/2}\cdot\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a\\
 & = & \frac{1}{2\cdot\left\Vert X\cdot a\right\Vert }\cdot\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a
\end{eqnarray*}
where 
\begin{eqnarray*}
 &  & \frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a=\\
 & = & \lim_{t\rightarrow0}\frac{a^{T}\cdot\left(X+t\cdot D\right)^{T}\cdot\left(X+t\cdot D\right)\cdot a-a^{T}\cdot X^{T}\cdot X\cdot a}{t}\\
 & = & \lim_{t\rightarrow0}\frac{a^{T}\cdot\left(X^{T}+t\cdot D^{T}\right)\cdot\left(X+t\cdot D\right)\cdot a-a^{T}\cdot X^{T}\cdot X\cdot a}{t}\\
 & = & \lim_{t\rightarrow0}\frac{t\cdot\left(a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a+t\cdot a^{T}\cdot D^{T}\cdot D\cdot a\right)}{t}\\
 & = & \lim_{t\rightarrow0}a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a+t\cdot a^{T}\cdot D^{T}\cdot D\cdot a\\
 & = & a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a\\
 & = & 2\cdot\left\langle X\cdot a,D\cdot a\right\rangle \\
 & = & 2\cdot Tr\left(a^{T}\cdot X^{T}\cdot D\cdot a\right)\\
 & = & 2\cdot Tr\left(a\cdot a^{T}\cdot X^{T}\cdot D\right)\\
 & = & 2\cdot\left\langle \left(a\cdot a^{T}\cdot X^{T}\right)^{T},D\right\rangle \\
 & = & 2\cdot\left\langle X\cdot a\cdot a^{T},D\right\rangle \\
 & = & \left\langle 2\cdot X\cdot a\cdot a^{T},D\right\rangle 
\end{eqnarray*}
and so you have $\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a=2\cdot X\cdot a\cdot a^{T}$.
Therefore your solution is the same as my solution:
\begin{eqnarray*}
\frac{\partial}{\partial X}\left\Vert X\cdot a\right\Vert  & = & \frac{X\cdot a\cdot a^{T}}{\left\Vert X\cdot a\right\Vert }
\end{eqnarray*}
By the way: The solution of the mathematicians I cannot understand,
too.
A: If the following is correct, then how are the above answers related to it?
$$
\frac{\partial}{\partial X}\left[\|Xa\|_2\right] = \frac{\partial}{\partial X}\left[(a^TX^TXa)^{\frac{1}{2}}\right]
= \frac{1}{2}(a^TX^TXa)^{-\frac{1}{2}}\cdot\frac{\partial}{\partial X}\left[(a^TX^TXa)\right]
=\frac{1}{2}(a^TX^TXa)^{-\frac{1}{2}}X(aa^T + aa^T)
= (a^TX^TXa)^{-\frac{1}{2}}Xaa^T
= \frac{Xaa^T}{\|Xa\|_2}
$$
