For me this type of question I have not ever seen it, my understanding to the solution is not clear, and I do not understand why the other choices are wrong, could anyone explain this for me please? Also any recommendation for books containing this type of questions would be greatly appreciated.
The group of symmetries of the pentagram consists of all reflections and rotations mapping the pentagram to itself. Any rotation doing this will map the regular pentagon to itself (notice the small regular pentagon in the middle of the figure). Also, any reflection doing this must be a reflection about a line cutting the figure into two congruent pieces. Obviously, any such line will cut the pentagon into two congruent pieces as well. This "proves" that the symmetry group $G$ of the pentagram is isomorphic to a subgroup of $D_5$, the symmetry group of the pentagon (by mapping each symmetry of the former to its restriction to the pentagon). By looking at the diagram we can detect at least $10$ elements of $G$, so $G$ is actually isomorphic (equal) to $D_5$.
Hopefully this was sufficiently non-hand-wavy.
The automorphism group of the cycle graph $C_5$ is the Dihedral group of order 10. Now the pentagram is the graph complement of $C_5$. So the automorphism group is preserved.