Subgroups and invariants in a unitary group U(3) This is related to the post, but an enriched version of the problem. Now we require the richer form of $P_1,P_2,P_3,P_4,P_5,P_6$.
Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of U(3). Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrices.
Can we find some subgroup of Lie group,  $$k \in K \subset G= U(3) $$  such that

$$ 
k^T \{P_1, P_2,  P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6 \} k =\{P_1, P_2,  P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6\}.
$$
  This means that set $\{P_1, P_2,  P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6\}$ is invariant under the transformation by $k$. 
  Here $k^T$ is the transpose of $k$.
  What is the full subset (or subgroup) of $K$?

Here we define: 
$$
P_1 =
\left(
\begin{array}{ccc}
 0 & 1 & 0 \\
1 & 0 & 0 \\
 0 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_2 =
\left(
\begin{array}{ccc}
 0 & 0 & 1 \\
 0 & 0 & 0 \\
 1 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_3 =
\left(
\begin{array}{ccc}
 0 & 0 & 0 \\
 0 & 0 & 1 \\
 0 & 1 & 0 \\
\end{array}
\right).$$
$$
P_4 =\sqrt{2}
\left(
\begin{array}{ccc}
 1 & 0 & 0 \\
0 & 0 & 0 \\
 0 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_5 =
\sqrt{2}\left(
\begin{array}{ccc}
 0 & 0 & 0 \\
0 & 1 & 0 \\
 0 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_6 =
\sqrt{2}\left(
\begin{array}{ccc}
 0 & 0 & 0 \\
0 & 0 & 0 \\
 0 & 0 & 1 \\
\end{array}
\right).$$
This means that $k^T P_a k= \pm P_b$ which may transform $a$ to a different value $b$, where $a,b \in \{1,2,3,4,5,6 \}$. But overall the full set $ \{P_1, P_2,  P_3,P_4,P_5,P_6, -P_1, - P_2, - P_3,-P_4,-P_5,-P_6\}$ is invariant under the transformation by $k$.
There must be a trivial element $k=$ the rank-3 identity matrix. But what else can it allow? 

How could we determine the complete $K$?

 A: The answer (and the method) is the same as the previous question.

The subgroup $K$ of $U(3)$ containing invariant matrices are isomorphic to the finite group
  $$
\mathbb{Z}_4\times S_4 \cong\langle i\rangle\times D(2,3,4)
$$
  where $\langle i\rangle=\{\pm I,\pm iI\}\cong\mathbb{Z}_4$ and $D(2,3,4)$ is the von Dyck group which is isomorphic to $S_4$.
More specifically, $D(2,3,4)=\langle a,b,c \mid a^2=b^3=c^4=abc=I\rangle$ is represented in $U(3)$ as follows:
  $$
a = \begin{pmatrix} -1 & 0 & 0 \\ 0 & 0 & 1 \\ 0 & 1 & 0 \end{pmatrix}, \quad
b = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0 \end{pmatrix}, \quad
c = \begin{pmatrix} 0 & 1 & 0 \\ -1 & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}
$$



*

*It is a fact from elementary linear algebra that $P_a$ and $k^TP_ak$ have the same rank because $k\in U(3)$ is non-singular. Notice that $P_1,P_2,P_3$ are of rank $2$, but $P_4,P_5,P_6$ are of rank $1$. Thus, if $k^TP_ak=\pm P_b$, then either $a,b\in\{1,2,3\}$ or $a,b\in\{4,5,6\}$.

*All the matrices preserving $\{P_i\mid 1\leq i\leq 6\}$ also preserve $\{P_i\mid 1\leq i\leq 3\}$.

*Because we already know all the matrices preserving $\{P_i\mid 1\leq i\leq 3\}$ in the previous question, it suffices to check whether those matrices preserve $\{P_i\mid 4\leq i\leq 6\}$ or not.

*Note that the three generators $a,b,c$ preserve $\{P_i\mid 4\leq i\leq 6\}$ as follows
$$
\begin{gather*}
a^TP_4a=P_4, \quad b^TP_4b=P_5, \quad c^TP_4c=P_5 \\
a^TP_5a=P_6, \quad b^TP_5b=P_6, \quad c^TP_5c=P_4 \\
a^TP_6a=P_5, \quad b^TP_6b=P_4, \quad c^TP_6c=P_6
\end{gather*}
$$

*It is trivial that $\langle i\rangle=\{\pm I,\pm iI\}$ preserve $\{P_i\mid 4\leq i\leq 6\}$.

*Therefore we have the same solution as the previous question.
