What is symmetric about the symmetric group? In my abstract algebra reader many times they use the word "symmetry". 
But I just found out that I'm not quite sure what they mean with it.

Let $X=\{1,...,n\}$. The set $S_n$ of bijections $X\to X$ under
  composition with identity $X\to X :x\mapsto x$  is called the
  symmetric group on $n$ symbols.

If I think about symmetry I think about figures, as something I can visualize. However I'm not sure if or how I should visualize the symmetric group of $n$ symbols as something symmetric. 
Edit: Thanks for all the answers! They all contributed to be finally satisfied with calling $S_n$ symmetric :)
Edit2: This helped me a lot as well: http://en.wikipedia.org/wiki/Frucht%27s_theorem

Frucht's theorem says that every finite group is the symmetry group of
  some graph. So every finite abstract group is actually the symmetries
  of some explicit object.

 A: Here is one way to think about this:
Indeed, we like to think of a group as the symmetries of some object, but what sort of symmetries do we mean?
For example, if we take a dihedral group, we are looking at the symmetries of some points, where we additionally require that certain lines are preserved (so we want bijective maps from the set consisting of those points to itself, such that these maps also preserve those lines).
For the symmetric group, we in some sense forget all about extra structures we might otherwise want, so we take all the symmetries of the points (ie, all bijective maps, with no additional restrictions).
Added: To reply to your question about the symmetry being about it not mattering if we relabel the elements: To some extend, but there is a bit more to it.
In some sense, the reason the elements we are permuting cannot be distinguished is that any of them can be "sent" to any other. But this is also the case for the dihedral groups. So both for the symmetric groups and for the dihedral groups, we cannot distinguish the individual points we are permuting.
What we can, however distinguish for the dihedral groups and still not for the symmetric groups are pairs of distinct points. This is because, given two pairs of distinct points, if the dihedral group can send one pair to the other, then those pairs must be connected in the same way (ie, by the same number of line segments in the regular polygon we have the dihedral group acting on). On the other hand, given any such two pairs, there is a permutation in the symmetrix group that sends one to the other.
The above is an example of something known as transitivity and $2$-transitivity. For more information about this, one can read my recent answer to Degree of a permutation group
A: You can present $S_n$ as the group of automorphisms of a complete graph with $n$ vertices.
A: The origin of symmetric group began with geometric idea. Given for example equilateral triangle $ABC$, we search all rotations and reflexions which leave invariant the triangle, so the rotations and reflexions which send a vertex to another. 
If we denote $A=1$, $B=2$ and $C=3$, a possible rotation is that send $1$ to $2$ and $2$ to $3$ and $3$ to $1$ hence we find the permutation:
$$\sigma=\left(\begin{array}{cc}1&2&3\\2&3&1\end{array}\right).$$
A: The symmetric group of $n$ symbols is the group of symmetries of the $n$-simplex, which is the most symmetric $n$-dimensional polytope. The alternating group $A_n$ is the subgroup that preserves orientation.
(The $n$-simplex is just an $n$-dimensional analog of the triangle or the tetrahedron).
