Why do we consider actions on vector spaces but not finite sets when doing representations of finite groups? Say we have a finite group $G$, why we define a representation of it being a linear action on a finite dimensional vector space instead of on a finite set? As when we try to define a linear action on $V$, essentially we are defining that from one set of basis elements to another, and this job can be done by considering some actions on finite set $S$ as well. Thanks.
 A: Linear actions are quite special, and in some sense are "easier" to study than actions on sets, since there is a lot more structure on a vector space and we have many more tools for dealing with vector spaces. I once heard someone say that "eventually we would like to understand group actions on finite sets, but for now we'll settle for representation theory".
Not every linear action arises from an action on a finite set, because not every linear operator is simply a permutation of basis vectors. For example, consider the action of the cyclic group $\mathbb{Z} / n\mathbb{Z}$ on a finite set containing a single point: it must be the trivial action. However, there is an action of $\mathbb{Z} / n\mathbb{Z}$ on the one-dimensional vector space $\mathbb{C}$ by sending a generator to $\exp(2 \pi i / n)$.
A: Joppys answer explains why representations on vector spaces are useful and that it is a mistake to think we can reduce actions on a vector space to permutations of the elements of a basis. I want to point out that actions on sets are more important than you seem to think.
Most of the time representation is taken to mean a representation as an action on a vector space, but that is not always the case. When we look at actions on a set, perhaps finite, we are thinking about permutation groups, about which a great deal is known. The best known examples of  representations of a group as permutations are the two regular representations, in which the group acts on its underlying set (i.e. the set of its elements) by multiplication on the left or on the right.
A particularly pleasing example of the use of a very similar permutation representation is the proof of the Sylow Theorems by considering the action by multiplication of a group $G$ of order $ p^{α}b $ (where $p$ is a prime) on the set $\Omega$ of its subsets of size $ p^{α} $, of which there are ${p^{α}b} \choose {p^α} $.
In particular,


*

*if $p$ does not divide $b$, a simple cancellation argument shows that ${p^{α}b} \choose {p^α} $ is not divisible by $p$;

*this means that some orbit $X$ of $\Omega$ under the action of $G$ has length not divisible by $p$;

*since the length an orbit containing an element $x$ is $ \frac {|G|} {|S_x|} $, (where $S_x$ is the stabiliser of $x$, i.e. the subgroup of elements which leave $x$ unchanged), for $X$, ${p^α}$ divides $ |S_x| $;

*since each element of $S_x$ takes a given element of the subset $x$ to a different element of $x$, $S_x$ has at most ${p^α}$ elements;

*hence $S_x$ is a subgroup of order ${p^α}$.


This proves the first Sylow Theorem, namely that if $|G|={p^α}b$ with $b$ coprime to $p$, $G$ has a subgroup of order ${p^α}$.
A: Even if you mostly care about actions of groups on finite sets, you quickly end up wanting to look at linear representations. The reason is that there are relationships between actions on finite sets that you can only "see" when passing to linear representations.
Here is a simple example:  Consider the action of $S_4$ on a cube acting via rotations.  We have several natural finite set actions coming out of this: $S_4$ permutes say the faces, edges, and vertices of the cube.  If we just look at these as finite sets with an $S_4$ action they are all non-isomorphic, and indeed there are no $S_4$ equivariant set maps between them. But of course these actions are all connected, after all they are just looking at different parts of the same cube.
If we pass to the linearized actions (e.g. $S_4$ acting on formal linear combinations of vertices of the cube) there are now relationships between these actions in the form of maps of representations: For example you can define a linear map by sending a vertex to the sum of all edges adjacent to it.
