Defining representations; representations and semidirect products Let $G$ be a group, $X$ a set. Defining an action $G\times X \to X$ is the same as defining a group morphism $\rho: G\to Sym(X)$, through the formula $g\cdot - = \rho(g)$. The morphism $\rho$ is called a permutation representation.
Now, in what little representation theory I have studied one replaces the set $X$ with a vector space $V$, then defining a group morphism $\rho: G \to GL(V)$ is the same as defining an action $G\times V \to V$ such that $g\cdot-$ is linear for every g, through the same formula $g\cdot - = \rho(g)$. The morphism $\rho$ is called a linear representation.
The first question is: on what other objects can I make my group act? What are the conditions for the previous constructions to make sense when picking an object $X$ from a given category, and picking $Aut(X)$ the set of isomorphisms of $X$?
Addenda: 1) and when is this fruitful? On what categories have actions been studied?
2) Since we can always define through a morphism such a categorial representation, when is it equivalent to the definition of the form $G\times X \to X$?
For the second question: the construction of the semidirect product of groups $N$, $Q$ involves a morphism $Q \to Aut(N)$. One says that $Q$ acts on $N$ by automorphisms. I just noticed why one says so, and it's because it's just another case of an "action with extra structure". In this case it's the same giving such a morphism than giving an action $Q \times N \to N$ such that $q\cdot -$ is a group morphism for every $q$. 
So the semidirect product is linked to representations in this way. The second question is, are there analogue constructions with other type of representations? What I mean is: making a group act on another group yields a new group (the semidirect product) which has as underlying set the direct product of both groups. Do actions over different categories yield new objects in a similar fashion?
 A: The answer to your first question is essentially anything. Given any group $G$ and an object $X$ in a category $C$, a representation of $G$ on $X$ is precisely a group homomorphism $G \to \text{Aut}(X)$. For example, we can and do make groups act on


*

*abelian groups,

*topological spaces such as manifolds,

*graphs,

*categories,


and so forth. The refinement of your first question is addressed by this MO question. 
If $C$ is a concrete category (a category of sets and structure-preserving maps between sets) then giving a group action of $G$ on $X$ is the same as giving a map $G \times X \to X$ satisfying certain properties, but in general it's not possible to make sense of the object $G \times X$ if $G$ is just an ordinary group. If you want a general definition of group actions along these lines, then I think the right thing to do is to talk about group objects in $C$ instead of groups and require that $C$ have, say, finite products. Then it ought to be true that an action of a group object in $C$ on an object in $C$ is the same as a morphism $G \times X \to X$ satisfying certain properties, where $\times$ is the product in $C$. 
For example, a group object in $\text{Top}$ is a topological group, and a group action of a topological group on a topological space is precisely a continuous map $G \times X \to X$ satisfying certain properties. In situations like this, this definition is actually preferable, since I think $\text{Aut}(X)$ cannot always be defined as a topological group in the correct way. 

As long as you are thinking about category theory at all, there are natural ways to generalize the notion of representation, namely to functors. A group $G$ can be regarded as a category with one object and a morphism $g$ for every $g \in G$ which compose according to the way they compose in $G$, and then a permutation representation of $G$ is nothing more than a functor $G \to \text{Set}$, a linear representation is nothing more than a functor $G \to \text{Vect}$, and so forth. 
So not only can one consider more general target categories, but one can also consider more general source categories, and this is an enormously fruitful idea even if we restrict ourselves to situations which look vaguely like group representations. For example it leads to quiver representations, Lawvere theories, topological quantum field theories... and of course the general idea of a functor is enormously fruitful as well.
A: For the first question: one situation in which other representations have been very useful is in finite group theory and finite geometry.  There is a very close connection between certain finite groups and certain finite geometries, and so both directions are useful.
Many simple groups were discovered from their (finite) geometric properties.  For instance the Matheiu groups can be understood as (derivatives of) automorphism groups of Steiner systems, the finite projective classical groups are all automorphism groups of finite geometries such as projective spaces, the sporadic group HS is the automorphism group of the Higman–Sims graph.
To understand some fancy projective planes, Dickson and Zassenhaus classified all finite near-fields.  Later Suzuki classified all Zassenhaus groups, which look a lot like the automorphism groups of projective planes, but also included an infinite family of new simple groups, the Suzuki groups.
Many investigations on the structure of permutation groups quickly inherit a geometric feel, and many investigations in finite geometry require delicate classifications.  Many of the papers of William Kantor represent these two ideas quite well.

For the second question, I think it depends on what you want to use the new construction for.  One nice effect of semi-direct products is it allows one to consider any group acting on any other group as if the first is acting by conjugation.  This means you can use strange things like Sylow's theorem in either the actor or the actee.  It seems like such an advantage is lost if the actor and the actee are different sorts of objects.
However, you can take a semi-direct product of a matrix group and a vector space, and this is quite powerful.  It allows you to use group theory to understand a representation, and representation theory to understand a group.  Again, it works out nicely because both the group and the vector space are matrices, so you have a common ground to work with both of them.
