Invariance of subspaces in the context of representation theory I am a french-speaking undergraduate physics student. Therefore, I do not know of the specific English terminologies involved in the following discussion. I have tried to check everything up, but I might have done some mistakes. That's why I am first going to define everything useful to my question.
Homomorphism. For any linear map $T$ from $X$ to $Y$, the map is said to be an homomorphism if for any $x_1,x_2\in X$, $T(x_1x_2) = T(x_1)T(x_2)$.
Representation. A representation $T$ of a group $G$ is an homomorphism from $G$ to $GL(V)$.
Invariant subspace. If $T$ is a representation of the group $G$ in the vector space $V$, $U\subset V$ is said to be an invariant subspace if $\forall g\in G,\forall v\in V, T(g)v\in U$.
Orthogonal subspace. Let $V$ be the vector space associated with the unitary representation $T$ and $W$ be the invariant subspace of $V$ through $T$. One defines the orthogonal subspace $W^\bot$ as {y : $x^\bot y = 0$, $\forall x\in W$}.
In my course, the prof shows the following property.
Property shown. Let V be the vector space associated to the unitary representation $T$ and let $W$ be a subspace of $V$. If $W$ is an invariant subspace of $V$ through $T$ then the associated orthogonal subspace (denoted $W^\bot$) is invariant as well.
I am pretty sure that one could make this a more general statement, by inserting an if and only if condition. I think I showed the property from the other side, but I would like for someone to review my proof.
Property I want to show. Let $V$ be the vector space associated with the unitary representation $T$ and let $W$ be a subspace of $V$. If $W^\bot$ is an invariant subspace of $V$ through $T$, then $W$ is invariant as well.
Proof. Let $x\in W$ and $y\in W^\bot$.
\begin{equation}
\forall g\in G, (T(g)x)^\dagger y = x^\dagger T^\dagger (g)y = x^\dagger T^{-1}(g)y
\end{equation}
The last equality holds true for any unitary operator.
\begin{equation}
x^\dagger T^{-1}(g)y = x^\dagger T(g^{-1})y = 0
\end{equation}
Where we used the fact that $W^\bot$ is an invariant subspace. We therefore have that $(T(x)x)^\dagger y = 0$, which means that $T(g)x\in W$ : we have shown that $W$ is an invariant subspace of $V$ through $T(g)$.
Is this proof correct? Moreover, can we, therefore, say that
Theorem. Let V be the vector space associated to the unitary representation $T$ and let $W$ be a subspace of $V$. Let $W^\bot$ be the associated orthogonal subspace. $W$ is invariant if and only if $W^\bot$ is invariant.
 A: Regardless of the language problem, there are several points in your question that I think is mathematically unclear. You have basically omitted all assumptions so that many things have to be guessed.
Homomorphism. What you want to define is a homomorphism between groups, thus $X, Y$ are groups and it doesn't make sense to say that $T$ is a "linear map".
Invariant subspace. The correct definition is $\forall g\in G,\forall v\in U,T(g)v\in U$.
Orthogonal subspace. When talking about "unitary representation", you are assuming implicitly that the vector space $V$ is equipped with an inner product $\langle \cdot, \cdot \rangle$. Assuming that $V$ is a $\Bbb C$-vector space (which is usually the case but not always), one usually needs to assume that

*

*the inner product is invariant under $T$, namely $\langle T(g)v_1, T(g)v_2\rangle = \langle v_1, v_2\rangle$ for all $g \in G$ ("unitary");

*the inner product is positive definite (or at least nondegenerate);

*in case $V$ is infinite dimensional, perhaps want to assume that $(V, \langle \cdot, \cdot\rangle)$ is a Hilbert space. In that case, you may also want to assume that $W$ is a closed subspace.

If these assumptions are indeed assumed, then the result you state is true, as it is a simple consequence of duality, namely $(W^\bot)^\bot = W$ and hence you can apply what your prof showed to $W^\bot$.
