Invariant subgroup in a self-conjugate pair multiplication in Unitary Group $U(3)$ Inspired by @ChoF and @annie heart's exploration, I like to post a similar but possibly richer structure related to the decompositions of $U(3)$ or $SU(3)$ representation multiplications $$3 \otimes \bar 3 = 1 \oplus  8.$$ 
3 is in the fundamental and 8 is in the adjoint representation of $SU(3)$. Below I modify and preserve the structure of questions from (Subgroups and invariants in a unitary group U(3))
Let $$G=U(3),$$ be the unitary group. Here we consider $G$ in terms of the fundamental representation of U(3). Namely, all of $g \in G$ can be written as a rank-3 (3 by 3) matrices. Now we take a set of $P$ matrices from Gell-Mann matrices. 
Can we find some subgroup of Lie group,  $$k \in K \subset G= U(3) $$  such that

$$ 
k^\dagger \{\pm P_1, \pm  P_2,\pm   P_3,\pm  P_4,\pm  P_5, \pm  P_6, \pm  P_7, \pm  P_8 \} k$$ 
  $$=\{\pm P_1, \pm  P_2,\pm   P_3,\pm  P_4,\pm  P_5, \pm  P_6, \pm  P_7, \pm  P_8\}.
$$
  This means that the full set $\{\pm P_1, \pm  P_2,\pm   P_3,\pm  P_4,\pm  P_5, \pm  P_6, \pm  P_7, \pm  P_8\}$ is invariant under the transformation by $k$. 
  Here $$k^\dagger \equiv (k^*)^T$$ is the complex conjugate ($*$) transpose ($T$) of $k$.
  What is the full subset (or subgroup) of $K \subset G$?

Here we define: 
$$
P_1 =
\left(
\begin{array}{ccc}
 0 & 1 & 0 \\
1 & 0 & 0 \\
 0 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_2 =
\left(
\begin{array}{ccc}
 0 & -i & 0 \\
 i & 0 & 0 \\
 0 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_3 =
\left(
\begin{array}{ccc}
 1 & 0 & 0 \\
 0 & -1 & 0 \\
 0 & 0 & 0 \\
\end{array}
\right)
$$
$$
P_4 =
\left(
\begin{array}{ccc}
 0 & 0 & 1 \\
0 & 0 & 0 \\
 1 & 0 & 0 \\
\end{array}
\right),\;\;\;\; P_5 =
\left(
\begin{array}{ccc}
 0 & 0 & -i \\
0 & 0 & 0 \\
 i & 0 & 0 \\
\end{array}
\right),\;\;\;\;$$ 
$$P_6 =
\left(
\begin{array}{ccc}
 0 & 0 & 0 \\
0 & 0 & 1 \\
 0 & 1 & 0 \\
\end{array}
\right),\;\;\;\; P_7 =
\left(
\begin{array}{ccc}
 0 & 0 & 0 \\
0 & 0 & -i \\
 0 & i & 0 \\
\end{array}
\right).$$
$$P_8 =\frac{1}{\sqrt 3}
\left(
\begin{array}{ccc}
 1 & 0 & 0 \\
0 & 1 & 0 \\
 0 & 0 & -2 \\
\end{array}
\right).$$
This means that $$k^\dagger P_a k= \pm P_b$$ which may transform $a$ to a different value $b$, where $a,b \in \{1,2,3,4,5,6,7,8 \}$. But overall the full set $ \{\pm P_1, \pm  P_2,\pm   P_3,\pm  P_4,\pm  P_5, \pm  P_6, \pm  P_7, \pm  P_8 \}$ is invariant under the transformation by $k$.

Please determine the complete $K$.

p.s. I am not so sure the $P_8$ is in the most symmetric form in accordance with $P_j$  $j \in \{1,2,3,4,5,6,7 \}$. So maybe one can suggest the possible modification of $P_8$ to have a larger $K \subset U(3)$
 A: Note that the transformation $k^\dagger P_a k$ preserves the eigenvalue information of $P_a$ so that it preserves trace, determinant, and rank, etc.


*

*Since only $P_8$ has different eigenvalues, we must have $k^\dagger P_8k=P_8$. Furthermore, any $k\in U(3)$ satisfying $k^\dagger P_8k=P_8$ has the form $\begin{pmatrix} U(2) & 0 \\ 0 & U(1) \end{pmatrix}$.

*Any $k\in U(2)\times U(1)$ transforms either (1st case)
$$
\{\pm P_4,\pm P_5\}\leftrightarrow\{\pm P_4,\pm P_5\} \quad\text{and}\quad
\{\pm P_6,\pm P_7\}\leftrightarrow\{\pm P_6,\pm P_7\}
$$
or (2nd case) $\{\pm P_4,\pm P_5\}\leftrightarrow\{\pm P_6,\pm P_7\}$.

*In the (1st case), $k$ has the form $\begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & \gamma \end{pmatrix}\in U(1)\times U(1)\times U(1)$, and
in the (2nd case), $k$ has the form $\begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & \gamma \end{pmatrix}\in U(1)\times U(1)\times U(1)$,
where $\alpha,\beta,\gamma\in U(1)$ satisfy
$$
(\alpha=\pm\gamma \text{ or} \pm i\gamma) \quad\text{and}\quad
(\beta=\pm\gamma \text{ or} \pm i\gamma)
$$

*Moreover, in both cases, $k$ transforms $\{\pm P_1,\pm P_2\}\leftrightarrow\{\pm P_1,\pm P_2\}$ and $\{\pm P_3\}\leftrightarrow\{\pm P_3\}$.



Answer. The invariant subgroup $K$ of $U(3)$ is isomorphic to the infinite group
  $$
K \simeq U(1) \times G
$$
  where $G = (\mathbb{Z}_4\times\mathbb{Z}_4) \rtimes_\varphi \mathbb{Z}_2$ (GAP ID [32,11]).
More specifically, for $\gamma\in U(1)$, the elements in $K$ are
  $$
\gamma \begin{pmatrix} \alpha & 0 & 0 \\ 0 & \beta & 0 \\ 0 & 0 & 1 \end{pmatrix}
\quad\text{or}\quad
\gamma \begin{pmatrix} 0 & \alpha & 0 \\ \beta & 0 & 0 \\ 0 & 0 & 1 \end{pmatrix}
$$
  where $\alpha,\beta\in\{\pm1,\pm i\}$.

