Definition of Aut(G) in the graph theory and group theory For a fixed group G, we define the collection of group automorphisms is the automorphism group Aut(G) in the group theory. (An automorphism: a permutation on the set G)
In the graph theory, on the other hand, the set of all automorphisms of a graph G is defined as Aut(G). (An automorphism: a vertices' permutation preserving adjacency)
In both cases, Aut(G) forms a permutation group.
Although a graph G itself is not a group, however, the definition of Aut(G) in the group theory is limited to a fixed group G.
Might it be possible to say "for a set G" instead of "for a group G?" The definition comes somewhat confusingly to me. Should we have to make a distinction for each case for this definition even though both theories are closely related to each other?
 A: There is definitely a concept of an automorphism group of a set. It's just the permutation group of that set (the set of all functions from that set to itself, under the operation of function composition). 
The following may or may not be helpful:
Most objects we study in math form these things called categories. The definition of a category is pretty simple, but very abstract. Essentially, a category is a collection of objects and "maps" between these objects called morphsims. 
For example, the category of groups has groups as objects and homomorphisms as morphisms. The category of sets has sets as objects and functions as morphisms.
Morphisms which have two-sided inverses are called isomoprhisms. An isomorphism between an object and itself is called an automorphism. For example, in the category of groups, the only morphisms which have two-sided inverses are the bijective ones, and so automorphisms are bijective homomorphisms from a group to itself. 
A very important property of morphisms is that some morphisms can be composed together (and this composition is associative).
Given any category $\mathcal C$ and an object $X$ in $\mathcal C$, we can for the automorphism group of $X$, denoted $\mathrm{Aut}(X)$, which is the set of all automorphisms of $X$ under composition.
Groups, graphs, and sets all form categories, so each of these objects has an automorphism group. The automorphism group $\mathrm{Aut}(X)$ really is a group, even if the object $X$ is not a group (as we've seen, it could be a set or a graph).
A: The set of automorphisms for any object $X$ forms a group. Since $f \in \operatorname{Aut}(X)$ is equivalently defined as (this is actually the definition in category theory) $\exists g : X \to X$ s.t.
$$f \circ g = g \circ f = \operatorname{id}_X$$
in other words, the morphism has an inverse. We thus have a group structure $(\operatorname{Aut}(X), \circ)$.

The answer to the last question is not really. Much of the tools in exists to generalize concepts and proofs to work for multiple situations at once. Thus, instead of requiring many proofs for different but related objects, mathematicians can use one proof that works for those objects.
