Derivative of the Euclidean norm with respect to matrix? If one defines the Euclidean norm of the matrix $A$ as follows
$$\|A\| = \sqrt{\mbox{tr}(A^H A)}$$
where $A^H$ is the conjugate transpose of $A$.
Then what would be the derivative of $\|A\|$ with respect to matrix $A$ itself? 
I am looking for a general formula. 
Actually, I have a solution from my professor but I couldn't find the reference to that. According to him,
$\frac{\partial ||A||}{A} = \frac{2A}{2\sqrt{tr(A^H A)}} = \frac{A}{||A||}$.
Is this solution valid or I am missing something?
 A: For ease of typing, I'll use the following notations 
$$\eqalign{
 X:Y &= {\rm tr}(X^TY)\cr
 f &= \|A\| \cr
 A^H &= (A^T)^*\cr
}$$
Treating $(A,A^*)$ independently, yields the Wirtinger derivatives as 
$$\eqalign{
 f^2 &= {\rm tr}(A^HA) = A^*:A \cr
 2f\,df &= A^*:dA \cr
 \frac{\partial f}{\partial A} &= \frac{A^*}{2f} \,\,\implies
 \frac{\partial f}{\partial A^*} = \frac{A}{2f} \cr
}$$
If $A\in{\mathbb R}^{m\times n}$, then the standard derivative process yields
$$\eqalign{
 f^2 &= {\rm tr}(A^TA) = A:A \cr
 2f\,df &= 2A:dA \cr
 \frac{\partial f}{\partial A} &= \frac{A}{f} \cr
}$$
which appears to be what your professor had in mind.
Update
Some more detail on the Wirtinger derivatives. 
The full differential contains terms for both $A^*$ and $A$
$$\eqalign{
 2f\,df &= A^*:dA + A:dA^* \cr
}$$
When $A^*$ is held constant, $dA^*=0$, leaving 
$$\eqalign{
 2f\,df &= A^*:dA \cr
 df &= \frac{A^*}{2f}:dA \cr
 \frac{\partial f}{\partial A} &= \frac{A^*}{2f} \cr
}$$
Conversely, if $A$ is held constant, then $dA=0$ and 
$$\eqalign{
 2f\,df &= A:dA^* \cr
 df &= \frac{A}{2f}:dA^* \cr
 \frac{\partial f}{\partial A^*} &= \frac{A}{2f} \cr
}$$
Finally, if $A$ is real then $A=A^*,\,$ $dA=dA^*,\,$ and
$$\eqalign{
 2f\,df &= 2A:dA \cr
 df &= A:dA \cr
 \frac{\partial f}{\partial A} &= \frac{A}{f} \cr
}$$
A: It simply does not exist. Let $A$ be $1\times1$ then what you want is a derivative like $\frac{d|z|}{dz}$ in essence. Which we know does not exist (I use a frescher's definition of derivative if you have a generalized alternative definition, let us know!)
A: Let us start with a simpler function,
$$G(X)  = \frac{1}{2}\|X\|^2$$
We then have,
$$G(X+V) = \frac{1}{2}\|X+V\|^2 = \frac{1}{2} tr((X+V)^H(X+V))=\frac{1}{2}(tr(X^HX)+tr(X^HV)+tr(V^HX)+tr(V^HV))\\
=G(X) + G(V) + tr(X^HV)$$
This gives us that the Frechet derivative of $G$ w.r.t. $X$ is $tr(X^HV)$. But we are really interested in the derivative of the function $F(X) = \|X\|$. Using the chain rule, we know that the derivative of $G$ is $\|X\|\cdot F'$ so we finally have $F' = \frac{X}{\|X\|}$
