What is the precise definition of a group action that is a symmetry?

I know this question may seem a bit silly and trivial, but I haven't found any precise definitions for what a "symmetry" is in my textbook for group theory or online and I feel like I don't know it rigorously enough. Specifically, the root of my question is:

Why doesn't the dihedral group of order 2n have n! elements?

For example, the symmetries of a square (that is the dihedral group of order $2n = 8$, 4 reflections and rotations) has order $8$, but why isn't, for example $(12)$ a valid element in it?

I understand that there are axes of rotations and symmetries, but is that it? Is there a more precise definition of what a "symmetry action" must constitute?

You've asked several questions. I'll start with the easy one.

If you imagine a square with vertices labeled $1,2,3,4$ (in cyclic order) there are just eight permutations of that set that respect the geometry of the square. $(1,2)$ isn't one of them - you can't interchange two adjacent vertices and leave the other two fixed. (You can do that with diagonally opposite vertices).

The second question asks for

a more precise definition of what a "symmetry action" must constitute

The precise definition depends on the structure whose symmetries you are trying to describe. Roughly speaking, its a bijective transformation that preserves all the properties that matter at the moment. For a square, those properties are the angles and the lengths of the sides. Then you prove geometric theorems that say the only operations with that property are the eight rotations and reflections.

The definition of a group and the theorems about groups have their roots in the struggle to make the idea of symmetry precise. Sometimes that history is lost when the study of abstract algebra begins with the definition.

Try looking at what that would do to the square, it would exchange two vertices, and remember you have to keep the edges attached to the vertices, so you would end up with an hourglass figure instead of a square, so it is not something that preserves the square shape (i.e. is a symmetry of the square shape). And that's what a symmetry is, it's a group action which leaves the set of points on the object invariant. This doesn't mean it leaves it invariant on every point, it can mix them up, but points in the set stay in the set (in this case your set is a square as a subset of eg. $\Bbb R^2$).