Why do we want *unitary* representations of locally compact groups into $B(H)$?

This is related to a previous question of mine but I have a more philosophical issue with the material. Everywhere I have looked for representations of locally compact groups into $B(H)$, everyone says that we want unitary representations but do not say what motivates this. Unitarity is definitely a nice property, but why should we hamper ourselves so much? Why not just map into $GL(H)$ (the invertible operators on $H$)? This is most closely aligned with the representation theory of finite groups and seems completely reasonable. I'm not looking for an "ends justify the means" kind of argument since when developing mathematics we don't even know what the "end" is necessarily but a more intuitive or lower level argument.

After thinking about this question for a few days, I've come to a few conclusions and observations which I will type here for posterity, I suppose.

1. In the finite and compact case, we can do a group averaging trick to make the representations unitary. This is not guaranteed in the locally compact setting.

2. In the finite dimensional case, we can develop the group ring. This is extremely useful in understanding the irreducible representations of the group since any irreducible representation appears in the left regular representation and the left regular representation is naturally occurring in the group ring. Formulating the group ring as the set of elements of the form $\sum_{g\in G}f(g)g$, we know that the product of two elements $\tilde{f} = \sum_hf(h)h$ and $\tilde{g}=\sum_hg(h)h$ is given by $\tilde{f}\tilde{g} = \sum_h\sum_{h'}f(h)g(h')hh' = \sum_h\left(\sum_{h'}f(h')g(h^{-1}h')\right)h$. We naturally see a convolution arise in this expression, so we can identify the functions $f$ and $g$ with elements of the $L^1(G)$ algebra.

This gives us two paths to go by when generalizing: we can either define the group ring of a locally compact group to be given by elements of the form $\int_G f(g)g\,dg$, where this integral is understood as a formal integral and $f\in L^1(G)$ or we can start by asking that the group ring consist of elements of the form $\int_G f(g)g$, where $f$ is continuous and of compact support (as this is the closest analogy we have to "finiteness" without ostensibly being zero). But since we want to create a Banach algebra from this set and convolution lies at the heart of the group ring, we would be forced to complete $C_c(G)$ with respect to the $L^1$ norm and we would end up with $L^1(G)$ again anyway. With this in mind, if one were to go through the machinery of looking at the representation lifted to the group ring as one does in the finite case, one would end up getting that if $\rho:G\rightarrow GL(H)$ for some Hilbert space $H$, then $\lVert\rho(g)\rVert\le M < \infty$ for all $g\in G$ in order to have meaningful operators on $B(H)$. Since the $\rho(g)$ are uniformly bounded in norm, taking cues from the finite and compact cases, we might as well ask that $\rho(g)$ be unitary.

Suppose $\rho:G\rightarrow GL(H)$ is a group homomorphism. One can somewhat run this thought process in reverse and ask the question: what if we work with general (formal) objects of the "group ring" given by

$$\mathbb{C}[G] = \left\{\int_Gf(g)g\,dg:\left\lVert\left(\int_Gf(g)\rho(g)\,dg\right)h\right\rVert<\infty\,\text{ for all }\,h\in H\right\}.$$

Here I have taken the approach that the formal object $\left(\int_Gf(g)g\,dg\right)h = \int_Gf(g)\rho(g)h\,dg$, akin to lifting the representation from the group to the group ring in the finite case. The restriction that $\left\lVert\left(\int_Gf(g)\rho(g)\,dg\right)h\right\rVert<\infty$ merely forces $\int_Gf(g)\rho(g)\,dg$ to be a well-defined operator on $B(H)$. By the uniform boundedness principle, I believe this is equivalent $\left\lVert\int_Gf(g)\rho(g)\,dg\right\rVert<\infty$. No real restrictions have been placed on either $f$ nor $\rho$ at this stage in terms of their analytic properties.

From this we can see that depending on how $\rho$ behaves, $f$ could have wildly varying behavior. For instance, consider $G = (\mathbb{R}^+,\cdot)$ and $\rho(x) = x^3I$. This is a representation but in order for $\left\lVert\int_Gf(g)\rho(g)\,dg\right\rVert<\infty$, $f$ has to have some somewhat restrictive decay properties. Likewise if we consider the opposite situation: let $\rho(x) = x^{-3}I$, again this is a representation but $f$ could have linear growth and still $\int_Gf(g)g\,dg$ would be a well-defined element in $\mathbb{C}[G]$.

So we see that without some restrictions on $\rho$, we cannot say anything meaningful about the group ring. Again, taking cues from the finite and compact cases, we might as well take $\rho$ to be unitary. Making this restriction nearly forces the $L^1(G)$ algebra to pop out naturally (upon considering some norm arguments and the like).

Note that people don't only study unitary representations. If $G$ is a real or $p$-adic Lie group, then the study of representations of $G$ is a big part of the theory of automorphic forms and representations (it is the local'' part of the theory), and one studies representations that are not necessarily unitary (or, perhaps better, not necessarily unitarizable).

In this context there is another adjective, admissible, discovered by Harish-Chandra, which is of crucial importance, and a key theorem of Harish-Chandra shows that topologically irreducible unitary representations are admissible.

In practice, the emphasis on unitary representations, to the extent it exists, probably arose because one of the first goals of rep. theory of locally compact groups was to generalize the Plancherel theorem from Fourier theory on locally compact abelian groups, and the Peter--Weyl theorem from the theory of compact groups, by determining the structure of $L^2(G)$ as a $G$-representation.

More generally, the theory of automorphic forms naturally leads to the question of determining the structure of $L^2(\Gamma\ G)$ as a $G$-representation, where $\Gamma$ is a discrete subgroup of $G$.

Since $L^2$ spaces are unitary, this leads to a focus on unitary reps.

You might find this post useful for obtaining more background on some of the motivations of the theory.