We can assume that $p$ is an odd prime. If $H$ is a subgroup of order $p$ (which exists, by Cauchy's theorem), it has index $2$ in $G$, so it must be normal. Indeed, if we write $G=H\sqcup gH$ then we cannot have $H=Hg$ for $1g=g\notin H$ by the first coset decomposition, so $G=H\sqcup Hg$ and $Hg=gH$. This works for any $g\notin H$, and is of course valid if $g\in H$, so $H$ is normal.
If $K$ is a normal subgroup of order two it has in particular order coprime to that of $H$, hence $H\cap K=1$. By cardinality considerations it follows that $HK=G$, and if $K$ is normal it follows that $G$ is the direct product of $H$ and $K$, so $G$ is isomorphic to $C_2\times C_p$, which is manifestly abelian. The argument above shows that any group of order $2p$ is isomorphic to either $C_2\times C_p=C_{2p}$ or to $C_2\ltimes C_p=D_{2p}$ the dihedral group of order $2p$. Note that any nontrivial semidirect product as above must be $D_{2p}$. Indeed, any nontrivial automorphism of order $2$ of $C_p$ sends a generator $r$ to some $r^i$ with $r^{i^2}=r$ (apply the automorphism again), or what is the same, $i^2=1\mod p$. This means that $i=1\mod p$ (which cannot happen since the identity has order $1$) or $i=-1\mod p$; that is $r\mapsto r^{-1}$, so the only nontrivial automorphism of $C_p$ of ordern $2$ is inversion.
The argument above can be modified as follows. Take $s$ an element of order $2$, $r$ an element of order $p$ in $G$, by Cauchy. Since the subgroup generated by $r$ is normal, $srs^{-1}=srs=r^i$ for some $i$, and raising to the $i$-th power we see that $sr^is= r^{i^2}$, but since $s^2=1$ we get $r^{i^2}=r$. Since $r$ has order $p$; it follows that $i^2=1\mod p$. If $i=1 \mod p$, $G$ is abelian for $srs^{-1}=r$, i.e. $sr=rs$, that is $r,s$ commute (and they generate all of $G$). Else we see that $i=-1\mod p$, so we get indeed $srs=r^{-1}$, which is the relation we wanted.
