Which categories are useful for representing groups? Given a group $G$ and a category $\mathcal{C}$, we may consider representations of $G$ in $\mathcal{C}$, i.e. homomorphisms $G \to \text{Aut}_{\mathcal{C}}(X)$, where $X$ is an object of $\mathcal{C}$.
Representation theory of groups is usually done in the settings when $\mathcal{C}$ is one of the categories $\text{Set}$, $\text{Vect}$ or $\text{Hilb}$ (the latter having isometries as morphisms). Are there any other categories in which representation theory of groups is done and yields valuable insights? Is it interesting for example to study the case when $\mathcal{C}$ is a category of modules, or a category of Lie algebras?
 A: Lots of thing have been done! This list is certainly incomplete:

*

*Geometric group theory studies groups through their action on topological and geometric spaces, generalizing the pemutation representations on sets.


*When studying representations on vector spaces it becomes more important to specify the precise category when working with infinite dimensional spaces (for non-compact Lie-groups this is quite natural). The reason is that when a group acts on a geometric space, it gives a linear representation on the vector space of functions on that space, but representations on smooth functions, $L^2$ functions, distributions etc all are different technically speaking but also dense inside each other and in a more meaningful way the same. (Infinitessimal equivalence)
The question is then what categories are useful to single out one representative of each equivalence class. You already mentioned Hilbert spaces, which are the place to go if you want unitary representations, but unitary reps are only a measure zero subspace of natural families of more general (inequivalent) representations on the same Hilbert space. If we think about the Hilbert space as $L^2$-functions on some geometric space then the representation on the smooth functions inside that Hilbert space can be in some context (e.g. the Casselman-Wallach theorem) a bit more useful, but this is a Frechet space and singling out this space comes down in the language of your question to representing the group in the category of Frechet spaces with some extra property.
Also the representation on its linear dual (the space of distributions on the geometric space if we stick with the picture of vectors as functions (or in this case generalized functions)) is also sometimes the one you want (when dealing with Payley-Wiener theorems in more general contexts if I recall correctly, but I don't know the details anymore). I believe it is a Banach space with some extra requirement, but I forgot all the details.

*

*Representatations on objects that have themselves a lot more structure (such as your Lie algebra example) are most viewed in the other direction: we have the object and wonder what its automorphism group is, rather than constructing the object in order to under stand the group. I am however aware of one notable counterexample: the monster Lie algebra that was constructed explicitly with the goal of having an explicit model of the monster group.


*As for modules: it is true that linear representation theory becomes different when working not over $\mathbb{C}$ but over some other ring or field. This has been studied for all sort of rings, I think the biggest area here is modular representation theory which studies representations on vector spaces over finite fields (so still over a field). However things start becoming really different when the characteristic of the field divides the order of the group. (So here we are back with finite groups again.)
