# Order of automorphism group

I have this tiny question that I just can't figure out: Let $G$ be the dihedral group of order 8. Show that Aut($G$) is a $2$-group.

I know that there is a general way to calculate the order of the automorphism group of a dihedral group, so the order in fact can be calculated to be $8$. But that is not how this question is supposed to be answered. Instead, I am given the hint that I can use the fact that the automorphism group of a cyclic $2$-group is a $2$-group.

I know that $G$ has a cyclic $2$-group. So I guess the way to proceed is to assume that Aut($G$) is not a $2$-group and derive a contradiction via some coprime action argument. But I am not completely sure how to do this.

Any ideas?

An important philosophy of group theory is to break up groups into their component pieces. This arises, for example, in extensions $H \to G \to G/H$ for normal subgroups $H$ of $G$, and motivates the search for simple groups. For the purposes of studying automorphisms, having a normal subgroup is good but not great (normal subgroups are fixed by inner automorphisms, but not outer ones, so you have to figure out how the outer ones can move the normal subgroup around).
Recall that a subgroup is characteristic if it is preserved by all automorphisms, not just the inner ones. Standard examples of characteristic subgroups include the center and the commutator subgroup. Suppose that $H < G$ is characteristic. Then one has a canonical homomorphism $\mathrm{Aut}(G) \to \mathrm{Aut}(H)$ given by restricting an automorphism of $G$ to its action just on $H$. The kernel of this homomorphism consists of those automorphisms of $G$ that fix $H$ pointwise. It follows that $|\mathrm{Aut}(G)| = \#\{$automorphisms of $G$ that fix $H$ pointwise$\} \times |$image of the homomorphism$|$. The latter factor necessarily divides $|\mathrm{Aut}(H)|$, as it is the size of a subgroup thereof.
Let $G = D_{2n}$ be a dihedral group of order $2n$, with $n\geq 3$. I claim that the subgroup $H = C_n$ of rotations is characteristic. Indeed, it is the unique proper subgroup containing an element of order $n$ (as all reflections have order $2$). Of course, $|\mathrm{Aut}(C_n)| = \phi(n)$ is Euler's totient function. When $n = 2^k$ for $k\geq 1$, $\phi(n) = 2^{k-1}$ counts the number of odd numbers less than $n$. This verifies your to-be-used statement that the automorphism group of a cyclic 2-group is a 2-group. It follows that the second factor in $|\mathrm{Aut}(G)|$ is a power of $2$ (since it is the size of a subgroup of a 2-group).
It remains to count the group of automorphisms of $D_{2n}$ that fix all rotations. There are various ways to do this count. The method that reduces to Babak Miraftab's answer is to observe that such an automorphism is determined by the image of a single reflection (as any one reflection, along with the rotations, generates the group), and that a given reflection can be sent to any other. Here is a less enlightening way to do the count. The group we care about is a subgroup of the symmetric group $S_4$ on the set of reflections. Since $|S_4| = 4! = 24$, it suffices to check that no automorphism has order $3$. The only permutations in $S_4$ of order $3$ cyclicly permute three things while leaving the last fixed. But the product of any two reflections is a rotation, and these we supposed are fixed; therefore the automorphisms we care about cannot fix a reflection without fixing all of them.
As you know $D_8=\langle a,b\,|a^4=b^2=1\rangle$. Let $f\in \operatorname{Aut}(D_8)$. Then, it is sufficient we define $f(a)$ and $f(b)$. Note that $\langle a\rangle$ is a characteristic subgroup of $D_8$-it is an unique subgroup. Since $o(b)=2$, we can conclude that $o(f(b))=2$ and moreover $f(b)\notin\langle a\rangle$. Hence, $f(b)=a^ib$ where $0\leq i< 4$. Thus, we can characterise every automorphism of $D_8$. Put $f_{i,j}(a)=a^j$ and $f_{i,j}(b)=a^ib$. Now, it is not hard to see that $\operatorname{Aut}(D_8)$ is $2$-group. One can see that the number of $j$ is $\phi(4)$ and the number of $i$ is $4$.