# Derivative of $\|Xa\|_2$ with respect to $X$

Can someone give me the answer to the following expression?

$\frac{\partial}{\partial X}\|Xa\|_2 =?$

$X$ is a square matrix and $a$ is a vektor of the apropriate size. $\|\cdot\|_2$ is the euclidean norm. Thanks

-
This one is handy. orion.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf –  user17762 Jun 13 '12 at 9:09
Think of the very-degenerate case, $X$ is a matrix with one entry, and we have $g(x,a)=|x\cdot a|$. Now we derive it with respect to $x$. The idea is the same, only in matrices. –  Asaf Karagila Jun 13 '12 at 9:10
To sum up: the accepted answer is wrong and the OP added another, equally wrong, answer. Cool. –  Did Jul 25 '12 at 16:45

Given the function \begin{eqnarray*} f\left(X\right) & = & \left\Vert X\cdot a\right\Vert \\ & = & \sqrt{a^{T}\cdot X^{T}\cdot X\cdot a}\\ & = & \left(a^{T}\cdot X^{T}\cdot X\cdot a\right)^{1/2} \end{eqnarray*} where $X$ is a matrix and $a$ is a vector. The gradient is \begin{eqnarray*} \frac{\partial f}{\partial X} & = & \frac{1}{2}\cdot\left(a^{T}\cdot X^{T}\cdot X\cdot a\right)^{-1/2}\cdot\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a\\ & = & \frac{1}{2\cdot\left\Vert X\cdot a\right\Vert }\cdot\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a \end{eqnarray*} where \begin{eqnarray*} & & \frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a=\\ & = & \lim_{t\rightarrow0}\frac{a^{T}\cdot\left(X+t\cdot D\right)^{T}\cdot\left(X+t\cdot D\right)\cdot a-a^{T}\cdot X^{T}\cdot X\cdot a}{t}\\ & = & \lim_{t\rightarrow0}\frac{a^{T}\cdot\left(X^{T}+t\cdot D^{T}\right)\cdot\left(X+t\cdot D\right)\cdot a-a^{T}\cdot X^{T}\cdot X\cdot a}{t}\\ & = & \lim_{t\rightarrow0}\frac{t\cdot\left(a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a+t\cdot a^{T}\cdot D^{T}\cdot D\cdot a\right)}{t}\\ & = & \lim_{t\rightarrow0}a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a+t\cdot a^{T}\cdot D^{T}\cdot D\cdot a\\ & = & a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a\\ & = & 2\cdot\left\langle X\cdot a,D\cdot a\right\rangle \\ & = & 2\cdot Tr\left(a^{T}\cdot X^{T}\cdot D\cdot a\right)\\ & = & 2\cdot Tr\left(a\cdot a^{T}\cdot X^{T}\cdot D\right)\\ & = & 2\cdot\left\langle \left(a\cdot a^{T}\cdot X^{T}\right)^{T},D\right\rangle \\ & = & 2\cdot\left\langle X\cdot a\cdot a^{T},D\right\rangle \\ & = & \left\langle 2\cdot X\cdot a\cdot a^{T},D\right\rangle \end{eqnarray*} and so you have $\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a=2\cdot X\cdot a\cdot a^{T}$. Therefore your solution is the same as my solution: \begin{eqnarray*} \frac{\partial}{\partial X}\left\Vert X\cdot a\right\Vert & = & \frac{X\cdot a\cdot a^{T}}{\left\Vert X\cdot a\right\Vert } \end{eqnarray*} By the way: The solution of the mathematicians I cannot understand, too.

-
By the way, the last formula is highly implausible and I have two questions: What is $D$? What is $\langle B,C\rangle$ for square matrices $B$ and $C$? –  Did Jun 14 '12 at 12:23

Take any matrix $B$ and expand $$\|(X+tB)a\|_2^2 = \|Xa\|_2^2 + 2 t \langle Xa , Ba \rangle + t^2 \|Ba\|_2^2.$$ Therefore the directional derivative of the square of your map in the direction $B$ is $$2 B^t Xa$$ where $B^t$ is the transposed matrix of $B$.

-
I am afraid I do not understand your answer. 2 remarks: 1.) I though taking derivative with respect to a matrix should give a matrix. Your expression gives a vector. 2.) I don't need the directional derivative. –  DK. Jun 13 '12 at 9:41
1) No, definitely no. Your function takes a matrix and produces a number. The derivative is therefore a vector, i.e. a gradient. 2) Please, first of all study the relationships between the derivative and the directional/partial derivatives. What I computed is the derivative of the square of your function, applied to a generic matrix $B$. –  Siminore Jun 13 '12 at 10:00

Recall that, when it exists, the differential $DF(X)$ at $X$ of a real-valued function $F$ defined on the space $M_n$ of the square matrices of size $n$ is the unique linear functional $L_X$ defined on $M_n$ such that $F(X+Y)=F(X)+L_X(Y)+\|Y\|_2\cdot\varepsilon_X(Y)$, where $\varepsilon_X(Y)$ has limit $0$ when $Y$ converges to the zero matrix.

Let $G:X\mapsto\|Xa\|_2^2$. Then $G(X)=a^tX^tXa$ and $G(X+Y)=G(X)+2a^tX^tYa+a^tY^tYa,$ hence $DG(X)$ exists and is given by $$DG(X)(Y)=2a^tX^tYa.$$ Let $F:X\mapsto\|Xa\|_2=\sqrt{G(X)}$. Assume first that $Xa\ne0$. Then, $$G(X+Y)=G(X)\cdot(1+F(X)^{-2}\cdot DG(X)(Y)+o(\|Y\|_2)).$$ Together with the fact that $\sqrt{1+t}=1+\frac12t+o(t)$ when $t\to0$, this yields $$F(X+Y)=F(X)\cdot(1+\tfrac12F(X)^{-2}\cdot DG(X)(Y)+o(\|Y\|_2)),$$ hence $$F(X+Y)=F(X)+\tfrac12F(X)^{-1}\cdot DG(X)(Y)+o(\|Y\|_2).$$ Thus, $DF(X)$ exists and is given by $$DF(X)(Y)=\tfrac12F(X)^{-1}\cdot DG(X)(Y)=\frac{a^tX^tYa}{\|Xa\|_2}.$$ The case $Xa=0$ is different. Now, $F(X+Y)=F(X)+\|Ya\|_2$.

• If $a=0$, $F$ is differentiable at $X$ and $DF(X)=0$ (in this case, $F=0$ everywhere).
• If $a\ne0$, $F$ is not differentiable at $X$ just like the function $t\mapsto|t|$ is not differentiable at $t=0$ (for example the function $Y\mapsto\|Ya\|_2$ is even, hence it cannot be linear at first order unless being zero).
-
Your answer is clear and rigorous. I like it. –  vesszabo Jun 13 '12 at 21:07
@vesszabo: Thanks for the appreciation. –  Did Jun 13 '12 at 21:12
I read somewhere that $\frac{\partial \det (A)}{\partial A}=\det(A) \left(A^{-1}\right)^T$. On the other hand $D(\det(X))=\det(X)\operatorname{tr}(X^{-1}DX)$. Hopefully I understand the last one, but I'm confused with the former one. (Let us assume that X is invertable, not this is my problem.) –  vesszabo Jun 14 '12 at 11:37

If the following is correct, then how are the above answers related to it?

$$\frac{\partial}{\partial X}\left[\|Xa\|_2\right] = \frac{\partial}{\partial X}\left[(a^TX^TXa)^{\frac{1}{2}}\right] = \frac{1}{2}(a^TX^TXa)^{-\frac{1}{2}}\cdot\frac{\partial}{\partial X}\left[(a^TX^TXa)\right] =\frac{1}{2}(a^TX^TXa)^{-\frac{1}{2}}X(aa^T + aa^T) = (a^TX^TXa)^{-\frac{1}{2}}Xaa^T = \frac{Xaa^T}{\|Xa\|_2}$$

-
The answer to your question is that what you call the following is not correct. For example, how does one transform $\frac{\partial}{\partial X}(a^TX^TXa)$ into $2Xaa^T$? –  Did Jun 14 '12 at 12:25