Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

Can someone give me the answer to the following expression?

$\frac{\partial}{\partial X}\|Xa\|_2 =?$

$X$ is a square matrix and $a$ is a vektor of the apropriate size. $\|\cdot\|_2$ is the euclidean norm. Thanks

share|improve this question
1  
This one is handy. orion.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf –  user17762 Jun 13 '12 at 9:09
    
Think of the very-degenerate case, $X$ is a matrix with one entry, and we have $g(x,a)=|x\cdot a|$. Now we derive it with respect to $x$. The idea is the same, only in matrices. –  Asaf Karagila Jun 13 '12 at 9:10
    
To sum up: the accepted answer is wrong and the OP added another, equally wrong, answer. Cool. –  Did Jul 25 '12 at 16:45
add comment

4 Answers

up vote 0 down vote accepted

Given the function \begin{eqnarray*} f\left(X\right) & = & \left\Vert X\cdot a\right\Vert \\ & = & \sqrt{a^{T}\cdot X^{T}\cdot X\cdot a}\\ & = & \left(a^{T}\cdot X^{T}\cdot X\cdot a\right)^{1/2} \end{eqnarray*} where $X$ is a matrix and $a$ is a vector. The gradient is \begin{eqnarray*} \frac{\partial f}{\partial X} & = & \frac{1}{2}\cdot\left(a^{T}\cdot X^{T}\cdot X\cdot a\right)^{-1/2}\cdot\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a\\ & = & \frac{1}{2\cdot\left\Vert X\cdot a\right\Vert }\cdot\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a \end{eqnarray*} where \begin{eqnarray*} & & \frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a=\\ & = & \lim_{t\rightarrow0}\frac{a^{T}\cdot\left(X+t\cdot D\right)^{T}\cdot\left(X+t\cdot D\right)\cdot a-a^{T}\cdot X^{T}\cdot X\cdot a}{t}\\ & = & \lim_{t\rightarrow0}\frac{a^{T}\cdot\left(X^{T}+t\cdot D^{T}\right)\cdot\left(X+t\cdot D\right)\cdot a-a^{T}\cdot X^{T}\cdot X\cdot a}{t}\\ & = & \lim_{t\rightarrow0}\frac{t\cdot\left(a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a+t\cdot a^{T}\cdot D^{T}\cdot D\cdot a\right)}{t}\\ & = & \lim_{t\rightarrow0}a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a+t\cdot a^{T}\cdot D^{T}\cdot D\cdot a\\ & = & a^{T}\cdot X^{T}\cdot D\cdot a+a^{T}\cdot D^{T}\cdot X\cdot a\\ & = & 2\cdot\left\langle X\cdot a,D\cdot a\right\rangle \\ & = & 2\cdot Tr\left(a^{T}\cdot X^{T}\cdot D\cdot a\right)\\ & = & 2\cdot Tr\left(a\cdot a^{T}\cdot X^{T}\cdot D\right)\\ & = & 2\cdot\left\langle \left(a\cdot a^{T}\cdot X^{T}\right)^{T},D\right\rangle \\ & = & 2\cdot\left\langle X\cdot a\cdot a^{T},D\right\rangle \\ & = & \left\langle 2\cdot X\cdot a\cdot a^{T},D\right\rangle \end{eqnarray*} and so you have $\frac{\partial}{\partial X}a^{T}\cdot X^{T}\cdot X\cdot a=2\cdot X\cdot a\cdot a^{T}$. Therefore your solution is the same as my solution: \begin{eqnarray*} \frac{\partial}{\partial X}\left\Vert X\cdot a\right\Vert & = & \frac{X\cdot a\cdot a^{T}}{\left\Vert X\cdot a\right\Vert } \end{eqnarray*} By the way: The solution of the mathematicians I cannot understand, too.

share|improve this answer
    
By the way, the last formula is highly implausible and I have two questions: What is $D$? What is $\langle B,C\rangle$ for square matrices $B$ and $C$? –  Did Jun 14 '12 at 12:23
add comment

Take any matrix $B$ and expand $$ \|(X+tB)a\|_2^2 = \|Xa\|_2^2 + 2 t \langle Xa , Ba \rangle + t^2 \|Ba\|_2^2. $$ Therefore the directional derivative of the square of your map in the direction $B$ is $$ 2 B^t Xa $$ where $B^t$ is the transposed matrix of $B$.

share|improve this answer
    
I am afraid I do not understand your answer. 2 remarks: 1.) I though taking derivative with respect to a matrix should give a matrix. Your expression gives a vector. 2.) I don't need the directional derivative. –  DK. Jun 13 '12 at 9:41
    
1) No, definitely no. Your function takes a matrix and produces a number. The derivative is therefore a vector, i.e. a gradient. 2) Please, first of all study the relationships between the derivative and the directional/partial derivatives. What I computed is the derivative of the square of your function, applied to a generic matrix $B$. –  Siminore Jun 13 '12 at 10:00
add comment

Recall that, when it exists, the differential $DF(X)$ at $X$ of a real-valued function $F$ defined on the space $M_n$ of the square matrices of size $n$ is the unique linear functional $L_X$ defined on $M_n$ such that $F(X+Y)=F(X)+L_X(Y)+\|Y\|_2\cdot\varepsilon_X(Y)$, where $\varepsilon_X(Y)$ has limit $0$ when $Y$ converges to the zero matrix.

Let $G:X\mapsto\|Xa\|_2^2$. Then $G(X)=a^tX^tXa$ and $ G(X+Y)=G(X)+2a^tX^tYa+a^tY^tYa, $ hence $DG(X)$ exists and is given by $$ DG(X)(Y)=2a^tX^tYa. $$ Let $F:X\mapsto\|Xa\|_2=\sqrt{G(X)}$. Assume first that $Xa\ne0$. Then, $$ G(X+Y)=G(X)\cdot(1+F(X)^{-2}\cdot DG(X)(Y)+o(\|Y\|_2)). $$ Together with the fact that $\sqrt{1+t}=1+\frac12t+o(t)$ when $t\to0$, this yields $$ F(X+Y)=F(X)\cdot(1+\tfrac12F(X)^{-2}\cdot DG(X)(Y)+o(\|Y\|_2)), $$ hence $$ F(X+Y)=F(X)+\tfrac12F(X)^{-1}\cdot DG(X)(Y)+o(\|Y\|_2). $$ Thus, $DF(X)$ exists and is given by $$ DF(X)(Y)=\tfrac12F(X)^{-1}\cdot DG(X)(Y)=\frac{a^tX^tYa}{\|Xa\|_2}. $$ The case $Xa=0$ is different. Now, $F(X+Y)=F(X)+\|Ya\|_2$.

  • If $a=0$, $F$ is differentiable at $X$ and $DF(X)=0$ (in this case, $F=0$ everywhere).
  • If $a\ne0$, $F$ is not differentiable at $X$ just like the function $t\mapsto|t|$ is not differentiable at $t=0$ (for example the function $Y\mapsto\|Ya\|_2$ is even, hence it cannot be linear at first order unless being zero).
share|improve this answer
1  
Your answer is clear and rigorous. I like it. –  vesszabo Jun 13 '12 at 21:07
    
@vesszabo: Thanks for the appreciation. –  Did Jun 13 '12 at 21:12
    
I read somewhere that $\frac{\partial \det (A)}{\partial A}=\det(A) \left(A^{-1}\right)^T$. On the other hand $D(\det(X))=\det(X)\operatorname{tr}(X^{-1}DX)$. Hopefully I understand the last one, but I'm confused with the former one. (Let us assume that X is invertable, not this is my problem.) –  vesszabo Jun 14 '12 at 11:37
add comment

If the following is correct, then how are the above answers related to it?

$$ \frac{\partial}{\partial X}\left[\|Xa\|_2\right] = \frac{\partial}{\partial X}\left[(a^TX^TXa)^{\frac{1}{2}}\right] = \frac{1}{2}(a^TX^TXa)^{-\frac{1}{2}}\cdot\frac{\partial}{\partial X}\left[(a^TX^TXa)\right] =\frac{1}{2}(a^TX^TXa)^{-\frac{1}{2}}X(aa^T + aa^T) = (a^TX^TXa)^{-\frac{1}{2}}Xaa^T = \frac{Xaa^T}{\|Xa\|_2} $$

share|improve this answer
    
The answer to your question is that what you call the following is not correct. For example, how does one transform $\frac{\partial}{\partial X}(a^TX^TXa)$ into $2Xaa^T$? –  Did Jun 14 '12 at 12:25
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.