A problem of linear algebra (Symmetric and antisymmetric matrixs and diagonalization) Suppose that you have $G$ defined as a linear function of $F$ of a matrix $U$, $G=F(U)$, $G$ is invariant under rigid motions $(\theta^{t} F(U) \theta=F(\theta^{t} U \theta), \theta^{t} \theta= \text{Id})$ and $G$ is symmetric.
Prove that $G$ only depends on the symmetric part of $U$, $S(U)$, and there are two constants $\alpha$ and $\beta$ such that $G=\alpha \text{Trace}(S(U)) \text{Id} + 2\beta S(U)$
My attempt: $U= S+A$ where $S$ is the symmetric part of $U$ and $A$ is the antisymmetric part of $U$. So $F(U)=F(S+A)=F(S)+F(A)$ as $G=F(U)$ is symmetric we have that $F(S)$ and $F(A)$ have to be symmetric (I don't understand exactly why this is true). As F(A) is symmetric we have that $F(A)=F(-A)=F(A^{t})$ which implies that $A=A^{t}$ and $A$ is symmetric which is false. So $F(U)=F(S)$.
And the second part I don't know how to even get started. I think my thoughts are wrong so if you can get me right and help me with the second part.
Thank you very much.
 A: I haven't any relevant references at hand, but I'd bet a question like this must have been solved in the literature. Given that you are not familiar with linear algebra, here is an elementary approach that assumes very little beyond basics. We suppose the dimensions of the matrices are $n\ge2$. The trivial case $n=1$ is ignored. Let
$$
H_t=\pmatrix{\cos t&\sin t\\ \sin t&-\cos t},\ K=\pmatrix{0&-1\\ 1&0},\ D=\pmatrix{1&0\\ 0&-1}.
$$
These three matrices are real orthogonal. In the sequel, we denote a block-diagonal matrix of the form $\pmatrix{X&0\\ 0&Y}$ by $X\oplus Y$.


*

*It is straightforward to show that $(H_t\oplus I_{n-2})^\top A(H_t\oplus I_{n-2}) = -A$ for every $t\in\mathbb R$ if and only if $A$ is a scalar multiple of $K\oplus0$. Hence $F(K\oplus0)$ is a scalar multiple of $K\oplus0$. Yet the images of $F$ are supposed to be symmetric. Therefore $F(K\oplus0)=0$. As the set of all skew-symmetric matrices is spanned by matrices of the form $\theta^\top (K\oplus0)\theta$ where $\theta$ is real orthogonal, $F$ maps every skew-symmetric matrix to zero.

*One can also verify that a matrix $A$ simultaneously satisfies
\begin{cases}
(K\oplus I_{n-2})^\top  A (K\oplus I_{n-2})&=-A,\\
(D\oplus I_{n-2})^\top  A (D\oplus I_{n-2})&=A
\end{cases}
if and only if $A$ is a scalar multiple of $D\oplus0$. Hence $F(D\oplus 0)$ is a scalar multiple of $D\oplus0$. Let $F(D\oplus 0)=2\beta D\oplus 0$. Yet, the set of all traceless symmetric matrices are spanned by matrices of the form $\theta^\top (D\oplus0)\theta$ where $\theta$ is real orthogonal. Hence $F(A)=2\beta A$ whenever $A$ is traceless and symmetric.

*Finally, by considering real orthogonal matrices obtained from flipping exactly one diagonal element of $I$ to $-1$, and also matrices obtained from replacing a principal $2\times2$ submatrix of $I$ by a rotation matrix, one can show that $\theta^\top A\theta=A$ for every real orthogonal $\theta$ if and only if $A$ is a scalar multiple of $I$. Hence $F(I)$ is a scalar multiple of $I$. Let $F(I)=(n\alpha+2\beta)I$.

*Now, every matrix $A$ can be written as a linear combination of $I$, a traceless symmetric matrix and a skew symmetric matrix: $A=\frac{\operatorname{tr}(S(A))}nI + \left(S(A)-\frac{\operatorname{tr}(S(A))}nI\right) + (A-S(A))$. So, our conclusion follows from the results of the previous steps.

A: Some early observations using only elementary arguments:
Since $F$ is linear it suffices to determine its action on elementary matrices such as $e_{ij}$, the matrix with a $1$ in row $i$, column $j$ and zeros everywhere else.
Consider the antisymmetric input $A = e_{12} - e_{21}$.  Let $R_1$ be the identity matrix with the first diagonal element replaced by $-1$ (reflection through first coordinate).  We easily calculate that $R_1^T A R_1 = -A$.  We also have $R_1^T R = I$, so we may apply the invariance of $F$ to obtain $R_1^T F(A) R_1 = F(R_1^T A R_1) = F(-A) = -F(A)$ by linearity.
We now show that $F(A)$ is constrained by considering an arbitrary symmetric matrix $M$ that satisfies $R_1^T M R_1 = -M$ (I'm only using $M$ because it's more convenient to write than $F(A)$).  Let $i,j$ denote arbitrary indices that are $>1$.  Then $-M_{ij} = (R_1^T M R_1)_{ij} = M_{ij}$, so $M_{ij} = 0$.  Similarly, $-M_{11} = (R_1^T M R_1)_{11} = M_{11}$, so $M_{11} = 0$.
So far we know that most entries of $F(A)$ are $0$, the exception being $F(A)_{1i}$ and identically $F(A)_{i1}$ for $i>1$.  Repeating the above argument with $R_2$ yields that these remaining entries are $0$ except for possibly $F(A)_{12} = F(A)_{21}$.  I believe these can be eliminated by using the permutation $P_{12}$ which swaps rows $1$ and $2$: this too is orthogonal and satisfies $P_{12}^T A P_{12} = -A$, and the same type of calculation should yield that $M_{21} = -M_{12}$, but $M_{21} = M_{12}$ by symmetry of $M$.
This establishes that $F(A)=0$ for an elementary antisymmetric matrix.  The argument generalizes to other elementary antisymmetric matrices like $e_{ij} - e_{ji}$, whence by linearity to arbitrary antisymmetric matrices.  This gives us $F(A) = 0$ for all antisymmetric $A$, meaning that $F(U) = F(S)$ by linearity of $F$, i.e. the first part of the problem.
