How to compute the Gateaux's derivative of a smooth but not analytic function? Let $f(x)=\begin{cases}x^2~\mbox{if $x\geq 0$}\\ 0~\mbox{otherwise}\end{cases}$
Let $A, B$ be two Hermitian matrices. We treat $f$ as a matrix function where $f(A)$ can be obtained from $A$ by applying $f$ to all the eigenvalues of $A$.
My question is if the following derivative well defined. If yes, how to compute it?
$\frac{d}{dt}f(A+tB)|_{t=0}$
 A: $f$ is not smooth but only $C^1$. 


*

*Two easy cases: $A>0$.  If $U$ is in a neighborhood of $A$, then $f(U)=U^2$ and $Df_A(B)=AB+BA$.


$A<0$. If $U$ is in a neighborhood of $A$, then $f(U)=0$ and $Df_A=0$.
EDIT. Finally,  I have done all calculation again and I was wrong (there is a stupid mistake in my programming).


*In the other cases, $f$ has also a derivative. More generally, when $f\in C^1$, then $f$ has a derivative on the vector-space of symmetric matrices (and even on the normal matrices); beware: the result is false over $M_n$.


I give some examples: If $A\geq 0$, then $Df_A(B)=AB+BA$.
If $A\leq 0$, then $Df_A=0$.
The most difficult case is when $A$ has $>0$ eigenvalues AND $<0$ eigenvalues. Clearly, we may assume that $A$ is diagonal.
For example, let $A=diag(\lambda_1,\lambda_2)=diag(-1,2)$. Then the $n^2=4$ eigenvalues of $Df_A$ are the $\dfrac{f(\lambda_i)-f(\lambda_j)}{\lambda_i-\lambda_j},(i,j)$, if $\lambda_i\not= \lambda_j$ and, otherwise $f'(\lambda_i)$; we obtain the symmetric matrix $A'=\begin{pmatrix}0&4/3\\4/3&4\end{pmatrix}$.
Finally, $Df_A(B)=A'\circ B$, where $"\circ"$ is the Hadamar product.
For the details, you can see: Higham, Functions of matrices, Theorem 3.11, p. 62.
