Classifying groups of order $20$

Now let $G$ be a nonabelian group of order $20$. By Sylow’s Theorem, $n_5 = 1$, so that $G$ has a unique (hence normal) Sylow $5$-subgroup $H \cong \mathbb{Z}_5$. Now let $K$ be any Sylow $2$-subgroup of $G$. By Lagrange, we have $H \cap K = 1$, so that $G = HK$. By the recognition theorem for semidirect products, $G \cong H \rtimes_\varphi K$ for some $\varphi : K \rightarrow \mathsf{Aut}(H)$. Evidently, classifying the nonabelian groups of order $20$ is equivalent to determining the nonisomorphic groups constructed in this manner. To that end, let $H = \mathbb{Z}_5 = \langle y \rangle$. Note that $\mathsf{Aut}(H) = \langle \alpha \rangle \cong \mathbb{Z}_4$; where $\alpha(y) = y^2$.

I don't understand the last sentence, the notation $\alpha(y)$ or why $\alpha(y) = y^2$ (how can a generator of a group be a mapping?) and why not $y^4$ (or some other arbitrary power).

• $S_4$ has more than $4$ elements, so it cannot be $S_4$. It is clear that $Aut(\mathbb{Z}_5)\cong \mathbb{Z}_4$, look at MSE questions about $Aut(\mathbb{Z}_n)$. Since $H$ is cyclic, it is enough to say, what $\alpha(y)$ should be. Oct 12, 2015 at 21:39

Note that any homomorphism of $\mathbb Z_5$ is determined by the image of a generator. It is an automorphism if that image is non-trivial. To be a little more explicit about things:
If $y$ is a generator, $\alpha$ is an automorphism and $\alpha(y)= y^2$ then $\alpha (y^r)=y^{2r}$
If we apply $\alpha$ twice, $\alpha^2(y)=y^4$, three times gives $\alpha^3(y)=y^8=y^3$ and $\alpha^4(y)=y^{16}=y$ so that $\alpha^4$ is the identity automorphism.
Note $S_{4}$ is a group of order $24$, not $4$. In fact, there are only two groups of order $4$ up to isomorphism: $\mathbb{Z}_{4}$, the cyclic group of order $4$, and $V_{4} \cong \mathbb{Z}_{2} \times \mathbb{Z}_{2}$, the so-called Klein 4-group. $\mathbb{Z}_{4}$ is distinguished from $V_{4}$ by the fact that it has an element of order $4$, whereas $V_{4}$ does not (every nonidentity element of $V_{4}$ has order $2$).
It is a well-known fact that $Aut(\mathbb{Z}_{n}) \cong (\mathbb{Z}/n\mathbb{Z})^{\times}$, but one doesn't need this in your specific case. The author of the post has supplied a generator for $Aut(H)$, namely the map (automorphism of $H$) $\alpha$ which sends the generator $y$ of $H$ to $y^{2}$. Since $Aut(H)$ is of order $4$, as you note, one must show that $\alpha, \alpha^{2}, \alpha^{3}, \alpha^{4}$ are distinct automorphisms. But this isn't too hard, as you can see that each of these maps takes $y$ to a distinct power of $y$. More precisely, $\alpha^{2}(y) = \alpha(\alpha(y)) = \alpha(y^{2}) = \alpha(y)^{2} = (y^{2})^{2} = y^{4}$. Via similar computations, one can show $\alpha^{3}(y) = y^{2^{3}} = y^{8} = y^{3}$, and $\alpha^{4}(y) = y^{2^{4}} = y^{16} = y$.