Rank of an action and definition of an orbital Let $G$ be a group acting on a set $X$. In group theory sometimes it is helpful to consider the action of $G$ on $X\times X$; a good example is perhaps finding the dimension of $\operatorname{Hom}(\sigma, \sigma)$ for a permutation representation $\sigma$ of the finite group $G$. I wonder why in this context it is always assumed that the action of $G$ on $X$ is transitive. In particular, the rank of an action is defined only when it is transitive. (See for instance Isaacs, Finite group theory page 257). Is there any important theorem which fails without transitivity?
I have also seen in another book (Steinberg, Representation Theory of Finite Groups Chapter 7) that if $G$ acts transitively on $X$, then for the corresponding permutation representation $\sigma$, we have
$$\dim \operatorname{Hom}(\sigma, \sigma)=\operatorname{rank}(\sigma).$$
I don't think this equality fails if the action is not transitive, but please correct me if I am missing something. (Of course, provided that we understand "rank of $\sigma$" on the RHS as the number of orbits of the action of $G$ on $X\times X$.)
 A: [I'm going to try to be a little agnostic about what a permutation representation is, because you seem to be implicitly passing to a field ($\mathbb{C}$?), though they can be studied in their own right as the category of morphisms $G\to\operatorname{Aut}(X)$.]
I think the point is that a non-transitive action gives you a representation that decomposes into the representations given by the (transitive) action on the orbits.  So transitive representations play a similar (but not identical) role that simple representations play in the theory of matrix representations.

Whether we focus on transitive actions, then, seems to be largely a question of our goals.  I could make various arguments in favor of either ignoring non-transitive permutations, or including them.
For example, it is very, very important, when working with general matrix representations, to think about representations that are not reducible.  Why?  Because most finite groups don't have a faithful irreducible representation.
But this issue doesn't apply to the category of permutation representations, since every finite group has a faithful transitive permutation representation, namely the regular representation(s).  So if our goal is to develop tools for describing finite groups using permutations, it's not obvious that we lose anything by restricting to transitive ones.

However, suppose—and I'm just making this up, though maybe it's a reasonable thing—that we want to prove something like Tannaka-Krein duality.  It's a nice theorem that we can recover a finite group $G$ from the category of its (matrix) representations over $\mathbb{C}$, if we include the structure of $\otimes$.  Specifically, I recall that $G$ is supposed to be the group of automorphisms of the identity functor that respect $\otimes$.  This tells you exactly why the character table of $G$ doesn't uniquely determine $G$, and what information to add so that it does.
If we wanted to do something similar for permutation representations, we need to be able to multiply two representations.  But if $\sigma$ is a representation of $G$ on $X$, then the only thing to reasonably call $\sigma\otimes\sigma$ is the obvious representation of $G$ on $X\times X$.  But then $\sigma\otimes\sigma$ is transitive exactly when $\sigma$ is doubly-transitive, so we'll have to go back and make sure we've proven something about non-transitive actions.

One last observation.  It is more difficult to pass from transitive permutation representations to general ones than it is to pass from simple matrix representations to general ones, for the following reason: There is no obvious notion of complementation, so there can be nontrivial morphisms (even arising from functions on the underlying sets!) between transitive representations—in other words, Schur's lemma fails, and dealing with decompositions becomes more difficult than in the case of matrix representations.  Even when $G$ is simple, there are some technicalities.
So this is a purely practical argument why an author might choose to avoid non-transitive representations altogether, if the option were available.

As for the specific theorem you mention: it does seem to be true for all actions, transitive, or no.  From Burnside's lemma, the number of orbits of $G$ acting on $X\times X$ is the average number of fixed points.  But if $\chi(g)$ is the number of fixed points of $g$ on $X$, then $\chi(g)^2$ is the number of fixed points on $X^2$.  But $\chi$ is the character of the representation over $\mathbb{C}$, so the number of orbits on $X\times X$ equals the inner product of $\chi$ with itself.  The result follows from Schur's lemma.
But something more subtle is going on here, and it's much easier to see when the action is transitive.  Fix an $x\in X$.  Then any choice of $y\in X$ gives us a unique orbit on $X\times X$ containing $(x,y)$, and all orbits arise this way.
Each orbit may contain multiple pairs, but this seems to give a dictionary between orbits and a nice basis for $\operatorname{End}(\sigma)$.  For example, take the regular representation of $S_3$.  Then $\operatorname{End}(\sigma)$ contains the map determined by  $e_1\mapsto e_1$, but also the map determined by $e_1 \mapsto e_2 + e_3$.  And $\{(1,1)\}$ and $\{(1,2), (1,3)\}$ are exactly the equivalence classes mentioned above.
You can give a description for a non-transitive action just as well, but it feels easier to come up with this kind of thing if you enforce transitivity.  Plus, looking at decomposition encourages looking for similar bijections related to $\operatorname{End}(\sigma,\tau)$, etc.
Anyway, mostly what I'm saying is that this seems to come down to a combination of goals and taste.  I would have to look much more deeply into the various authors' work to say more.
