Is there a way to simply this (vectorized matrix, derivative of a matrix) $vec^T((X^{-1})^2)^T\cdot|X^2|$
I got this as a part of
$\frac{d}{dX}|X^2|=vec^T((X^{-1})^2)^T\cdot|X^2|\cdot2X$
I'm afraid that the dimensions don't match so I might've done something wrong.
 A: Let $Y=X^2$, then it appears that you wish to calculate the gradient of the Frobenius Norm of $Y$ (which I'll denote by $N$) with respect to $X$. 
First, express the norm in terms of the Frobenius Product (denoted by a colon) and find the differential
$$\eqalign{
N^2 &= Y:Y \cr
2N\,dN &= 2Y:dY \cr
  &= 2Y:(X\,dX + dX\,X) \cr
  &= 2(X^TY+YX^T):dX \cr\cr
}$$
Next, find the gradient
$$\eqalign{
\frac{\partial N}{\partial X} &= \frac{1}{N}(X^TY+YX^T) \cr
  &= \frac{1}{N}(X^TX^2+X^2X^T) \cr\cr
}$$
If $X$ has any special properties, e.g. $X^T=X$, they can be used to simplify the result further.
Finally, you can (if you wish), vectorize both sides of the gradient.  But the right-hand-side will be very messy, involving Kronecker products and a Kronecker permutation matrix (due to the presence of $X^T$).
Update
It occurs to me that your question might have been about the determinant, instead of the norm. In which case, the function, differential, and gradient are a bit simpler
$$\eqalign{
 f &= \det(X^2) \cr
   &= \det(X)^2 \cr\cr
df &= 2\,\det(X)\,\,d\det(X) \cr
   &= 2\,\det(X)\,X^{-T}:dX \cr\cr
\frac{\partial f}{\partial X} &= 2\,\det(X)\,X^{-T} \cr
   &= 2\,{\rm adj}(X^T) \cr
}$$
where ${\rm adj}(A)$ denotes the adjugate of the matrix $A$.
