Derivative of $F^TF$ with respect to $F$ I stacked with evaluating the derivatives
$$
\frac{\partial\left(F^TF\right)}{\partial F}~~ {\rm and}~~ \frac{\partial^2\left(F^TF\right)}{\partial F^2}
$$
in terms of $F\in\mathbb{R}^{3\times3}$. Any hint or any idea would be acknowledged.
 A: The best way to approach this sort of problem is to use differentials rather than the chain rule.  
This is because the chain rule requires intermediate quantities which are 3rd & 4th order tensors, which are difficult to work with, whereas differentials are ordinary vector and matrix quantities.
For convenience, let $$\eqalign{M=F^TF-I\cr\cr}$$
Use the Frobenius (:) Inner Product and this new variable to rewrite the function and find its differential and gradient
$$\eqalign{
 W &= \frac{1}{2}\|M\|^2 \cr
   &= \frac{1}{2}M:M \cr\cr
dW &= M:dM \cr
   &= M:\big(\,dF^TF+F^TdF\,\big) \cr
   &= (FM^T+FM):dF \cr
   &= 2\,FM:dF\cr
   &= 2\,(FF^TF-F):dF \cr\cr
\frac{\partial W}{\partial F} &= 2\,(FF^TF-F) \cr\cr
}$$
Now let's proceed from the gradient to the the hessian (second derivative). Note that the gradient is a matrix, 
$$G=2\,(FF^TF-F)$$
so the hessian will be a 4th order tensor
$${\mathbb H} = \frac{\partial G}{\partial F}$$
Let's start with the differential of $G$
$$\eqalign{
dG &= 2\,(dF\,F^TF+F\,dF^TF+FF^TdF-dF) \cr\cr
}$$
To simplify further, we need 2 special (isotropic) tensors $({\mathbb E},{\mathbb B})$ with components
$$\eqalign{
{\mathbb E}_{ijkl} &= \delta_{ik}\,\delta_{jl}\cr
{\mathbb B}_{ijkl} &= \delta_{il}\,\delta_{jk}\cr
}$$
Then
$$\eqalign{
dG &= 2\,({\mathbb E}F^TF+F{\mathbb E}F^T:{\mathbb B}+FF^T{\mathbb E}-{\mathbb E}):dF \cr
{\mathbb H} &= 2\,({\mathbb E}F^TF+F{\mathbb E}F^T:{\mathbb B}+FF^T{\mathbb E}-{\mathbb E}) \cr\cr
}$$
Finally, you wished to evaluate the hessian at $F=I$, which simplifies things greatly
$$\eqalign{
{\mathbb H} &= 2\,({\mathbb E}+{\mathbb E}:{\mathbb B}+{\mathbb E}-{\mathbb E}) \cr
 &= 2\,({\mathbb E}+{\mathbb B}) \cr
}$$
A: Merging of two answers:
First a remark about derivatives in the algebra $M_3$ of matrices of size $n\times n$.
You need to give a meaning to $\frac{\partial F}{\partial F}$. The upper $F$ is a function of the $n^2$ elements of $F$ and you are calculating a derivative w.r.t. all the $n^2$ elements of $F$. So a way to write is
$ \frac{\partial F_{ij}}{\partial F_{kl}}=\delta_{ij,kl}$  which has $n^2\times n^2$ components. Now the derivative  in the direction $H\in H$ is then $\left( \frac{\partial F}{\partial F} H \right)_{ij} = \sum_{kl} \delta_{ij,kl}H_{kl} = H_{ij}$ an extremely cumbersome way of writing.
Should really be avoided if possible.
Setting $X=M_{3}({\Bbb C})$, $f:X \rightarrow X$,  $f(F)=F^T F$:
$$ f(F+H)= (F+H)^T (F+H) = F^T F + (F^T H + H^T F) + H^T H$$
which is identified with  $f(F) + f'(F).H + 1/2 f''(F).(H,H) + \ldots$
where $f'(F).H = F^T H + H^T F$ is the 1st derivative (acting upon $H$) and $f''(F).(H,K) = H^T K + K^T H$ is the second derivative (symmetric bilinear form). 
Now  a shortcut (if you don't worry too much about rigour?) to get the derivative of a differentiable function $W$ at $F$ in a direction $H$ all you need to calculate: $\frac{\partial W(F)}{\partial F} (H)=\frac{d}{dt}_{|t=0} W(F+tH)$.  Here $t$ is a just a usual real variable.  
Let us take (from the commentary rather than the question, which should perhaps be reformulated?) $W(F)=\frac{1}{4} |F^T F-I|^2=\frac{1}{4} {\rm tr} ((F^T F-I)(F^T F-I))$ with
 $F\in X=M_{3}({\Bbb C})$. Let us find the derivative in a direction $H$:
$$ \frac{\partial W(F)}{\partial F}(H)= \frac{d}{dt}_{|t=0} W(F+tH) = ... = \frac{1}{2} {\rm tr} ((H^T F + F^T H) (F^T F-I))$$
The second derivative of this expression in the direction $K$ is:
$$ \frac{\partial^2 W(F)}{\partial F^2}(H,K)= \frac{d}{dt}_{|t=0}\frac{\partial W(F+tK)}{\partial F}(H)  = ... = \frac{1}{2} {\rm tr} ((K^T H + H^T K) (F^T F-I))+
\frac{1}{2} {\rm tr}( (H^T F + F^T H)(K^T F + F^T K) )$$
Setting $F=I$ and $K=H$ the first term vanishes and the result is: 
$$ \frac{\partial^2 W(F)}{\partial F^2}(H,H)_{|F=I} =\frac{1}{2}  
{\rm tr}( (H^T + H)(H^T + H) ) = \frac{1}{2} |H^T+H|^2.$$
 Thus,
$$ \frac{\partial^2 W(F)}{\partial F^2}_{|F=I} (F,F) =  
\frac{1}{2} |F^T+F|^2$$
A: Hint: Expand using the product rule. It should be obvious how to simplify one of the resulting terms, so the problem will then be reduced to computing ${\partial(F^T)\over\partial F}$. One you have that, computing the second derivative should be easy.  
As another hint, consider the analogous problem in elementary calculus. what would you guess the second derivative of $F^TF$ ought to be?
