Invariants of a Reynolds stress tensor I am dealing with a Reynolds stress tensor, which arises in Navier-Stokes equations and is symmetric. By using an Ultrasound based instrument, I can read the velocity components of the fluid along three non orthogonal axis named 1-2-3. I can transform the measured velocities along these three axis in a Cartesian three orthogonal axis system x-y-z by writing the transformation matrix A and hence:
$\begin{Bmatrix}
u\\ 
v\\ 
w
\end{Bmatrix}=
\mathbf{A}\cdot \begin{Bmatrix}
u_1\\ 
u_2\\ 
u_3
\end{Bmatrix}                       $
where $\begin{Bmatrix}
u\\ 
v\\ 
w
\end{Bmatrix}$ is the velocity vector in a Cartesian orthogonal coordinate system, $u_1,u_2$ and $u_3$ are the three measured velocities along the three non orthogonal axes, $\mathbf{A}$ is the transformation matrix. The tensor transformation is:
$\mathbf{Q}^*=\mathbf{A}\cdot\mathbf{Q}\cdot\mathbf{A}^T$.
where $\mathbf{Q}$ is the Reynolds stress tensor as computed in the 1-2-3 instrument coordinate system:
$\mathbf{Q}=\begin{bmatrix}
 u_1^{\prime}u_1^{\prime}&u_1^{\prime}u_2^{\prime}  &u_1^{\prime}u_3^{\prime} \\ 
  u_2^{\prime}u_1^{\prime}&u_2^{\prime}u_2^{\prime}  &u_2^{\prime}u_3^{\prime} \\  
  u_3^{\prime}u_1^{\prime}&u_3^{\prime}u_2^{\prime}  &u_3^{\prime}u_3^{\prime} 
\end{bmatrix}$
and $\mathbf{Q}^*$ is the Reynolds stress tensor in the x-y-z coordinate system:
$\mathbf{Q}=\begin{bmatrix}
 u^{\prime}u^{\prime}&u^{\prime}v^{\prime}  &u^{\prime}w^{\prime} \\ 
  v^{\prime}u^{\prime}&v^{\prime}v^{\prime}  &v^{\prime}w^{\prime} \\ 
   w^{\prime}u^{\prime}&w^{\prime}v^{\prime}  &w^{\prime}w^{\prime}   
\end{bmatrix}$
The prime indicates that that the variable is the fluctuating part and the overline indicates the mean value, e.g. $u=\overline{u}+u^{\prime}$.
One half of the trace of the tensor $\mathbf{Q}^*$ is the turbulent kinetic energy of the fluid (per unit mass) and should be invariant. Indeed it is invariant as long as the transformation matrix $\mathbf{A}$ is orthogonal, i.e. $\mathbf{A}^{-1}=\mathbf{A}^T$ but it is not for general transformation matrix. In fact the trace of the tensor $\mathbf{Q}$ is different from the trace of the tensor $\mathbf{Q}^*$ if $\mathbf{}$ is the non-orthogonal transformation matrix between the two coordinate system 1-2-3 and x-y-z.
I wonder if the invariants of a tensor are really invariant (i.e. assume the same value) only for specific class of transformations (e.g. for similarity transformations).
 A: The problem is that you are not defining "trace" correctly. 
The trace of a rank 2 tensor is only equivalent to the sum of the diagonal entries of its matrix representation if the tensor is type (1,1). (Type (1,1) here means that the tensor represents a linear transformation; or equivalently that the matrix representation transforms like $\mathbf{Q} \to A\mathbf{Q}A^{-1}$ under change of basis/coordinates in the vector space.)
The Reynold stress tensor as you defined it is a type (0,2) (or type (2,0), depending on your convention) tensor, so its transformation law is $\mathbf{Q} \to A\mathbf{Q}A^T$. This type of tensors cannot be "traced"...until you specify what you are tracing with respect to. To be able to trace, you need to first multiply a type (0,2) tensor against a type (2,0) tensor to form a type (1,1) tensor, and then you can perform the trace. The type (2,0) tensor in this case is the symmetric bilinear form that corresponds to the Euclidean inner product. 

In other words, going back to the more physical part of the story: the kinetic energy is an invariant, but you are just computing it wrong. Think about the case where you have just a particle. The kinetic energy of a particle is $\frac12 m \vec{v}\cdot\vec{v}$. In an orthonormal coordinate system 
$$ \vec{v}\cdot \vec{v} = (v_x)^2 + (v_y)^2 + (v_z)^2 $$
but in a coordinate system that is not orthonormal, the dot product is not given by just the sum of the squares of the components! (For example, consider spherical or cylindrical coordinates!)
