Generalizing the case $p=2$ we would like to know if the statement below is true.
Let $p$ the smallest prime dividing the order of $G$. If $H$ is a subgroup of $G$ with index $p$ then $H$ is normal.
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It only takes a minute to sign up.
Sign up to join this communityGeneralizing the case $p=2$ we would like to know if the statement below is true.
Let $p$ the smallest prime dividing the order of $G$. If $H$ is a subgroup of $G$ with index $p$ then $H$ is normal.
This is a standard exercise, and the answer is that the statement is true, but the proof is rather different from the elementary way in which the $p=2$ case can be proven.
Let $H$ be a subgroup of index $p$ where $p$ is the smallest prime that divides $|G|$. Then $G$ acts on the set of left cosets of $H$, $\{gH\mid g\in G\}$ by left multiplication, $x\cdot(gH) = xgH$.
This action induces a homomorphism from $G\to S_p$, whose kernel is contained in $H$. Let $K$ be the kernel. Then $G/K$ is isomorphic to a subgroup of $S_p$, and so has order dividing $p!$. But it must also have order dividing $|G|$, and since $p$ is the smallest prime that divides $|G|$, it follows that $|G/K|=p$. Since $|G/K| = [G:K]=[G:H][H:K] = p[H:K]$, it follows that $[H:K]=1$, so $K=H$. Since $K$ is normal, $H$ was in fact normal.
Here is a slightly different way to prove the result:
We will do it by induction on $|G|$. If $G$ has just one subgroup of index $p$ then clearly that subgroup is normal, so let $H_1$ and $H_2$ be distinct subgroups of index $p$. We then have that $|H_1H_2|$ is a multiple of $|H_1|$, but due to the choice of $p$ we must in fact have $H_1H_2 = G$ which means that if we let $K = H_1 \cap H_2$ then $K$ has index $p$ in $H_1$ and $H_2$ so by induction we know that $K$ is normal in $H_1$ and $H_2$ and thus normal in $G$. Now we know that $G/K$ has order $p^2$ so it is abelian. Now since both $H_1$ and $H_2$ contain $K$ they correspond to subgroups of $G/K$ and since this is abelian, they correspond to normal subgroups, which shows that they are normal in $G$ as desired.
Hint: Consider the set of cosets $G/H$ of which there are $p$. Then $G$ acts on these cosets by left multiplication. So you have a homomorphism $\phi: G \rightarrow S_p$. If $p$ is the smallest prime dividing $|G|$ then what can you say about $|\mathrm{im} \phi|$ and what does this imply about $\ker \phi$?
proof: If $H$ is not normal then assume $H\neq H^g$ for some $g \in G$. But a classical formula says \begin{equation} |HH^g|=\dfrac{|H|\cdot |H^g|}{|H\cap H^g|} . \label{1} \tag{1} \end{equation}
Notice that $H\cap H^g$ is a proper subgroup of $H$ (proper since $H \neq H^g$). Hence, $|H\cap H^g|$ is a proper divisor of $|H|$. Since every prime divisor of $|H|$ is $\geq p$, this leads to $\dfrac{|H|}{|H\cap H^g|}\geq p$. Thus, \eqref{1} yields $|HH^g|\geq p|H^g| = |G|$. Therefore, $HH^g=G$.
Thus, $g=hg^{-1}kg$ for some $h,k\in H$. Therefore, $g=kh\in H$. As a result, $H=H^g$, which is a contradiction.
Here's yet another, slightly different proof:
Instead of considering the action of $G$ on the left cosets of $H$, let us consider the action of $H$ on left cosets of $H$.
By the orbit-stabilizer theorem, the size of every orbit of cosets divides $|H|$, and hence also $|G|$. Since there are exactly $p$ cosets of $H$ and $p$ is the smallest prime dividing $|G|$, it must be that either there is a single orbit of size $p$ or there are $p$ different orbits, all of size $1$.
The first option, however, is impossible, since for every $h\in H$, $hH=H$, meaning the action fixes the coset corresponding to the identity. Hence, there exists an orbit of size $1$, so they must all be of size $1$.
This means that for every $h \in H, g \in G$ we have: $$hg^{-1}H=g^{-1}H\implies \exists h_1\in H \space s.t.\space hg^{-1}=g^{-1}h_1\implies \\ ghg^{-1}=h_1\in H$$
And we are done.
Hint: Let $G$ act on $G/H$ by left multiplication. This gives you a homomorphism $G\to S_p$. Try to show that $H$ is the kernel of this map--note that if $q$ is a prime larger than $p$ then $q\nmid p!$.
Another answer if you know character theory over the complex numbers.
Let $H$ be a subgroup of order $G$ of index $p$, smallest prime dividing $|G|$. Look at the trivial character of $H$ and induce it to $G$. Then $1_H^G=\sum_{\chi \in{\rm Irr}(G)} a_{\chi}\chi$, with $a_{\chi} \in \mathbb{Z}_{\geq 0}$. Since $[1_H^G, 1_G]=[1_H,1_H]=1=a_{1_G}$, it follows that the irreducible constituents of $1_H^G$ have degree $\chi(1) \leq p-1 \lt p$. Since these degrees divide $|G|$ is follows that all the irreducible constituents $\chi$ must be linear, that is $\chi(1)=1$. Hence $H \supset \ker(1_H^G)=\cap\{{\ker(\chi) \in{\rm Irr}(G): a_{\chi} \gt 0}\} \supseteq G'$, whence $H \lhd G$.
In addition to the things answered here, I just want to add this one.
Proposition Let $G$ be a finite group and $H$ a subgroup of prime index $p$, with gcd$(|G|,p-1)=1$. Then $G' \subseteq H$.
Note that this implies that $H \unlhd G$, and that it is in fact sufficient to prove that $H$ is normal, since then $G/H \cong C_p$ is abelian.
Proof Firstly, we may assume by induction on $|G|$, that $H$ is core-free, that is core$_G(H)=\bigcap_{g \in G}H^g=1$. This means that $G$ can be homomorphically embedded in $S_p$. Let $P \in Syl_p(G)$ and note that because $|S_p|=p \cdot (p-1) \cdots \cdot 1$, $|P|=p$. By the $N/C$-Theorem, $N_G(P)/C_G(P)$ embeds in Aut$(P) \cong C_{p-1}$. By the assumption gcd$(|G|,p-1)=1$, we get that $N_G(P)=C_G(P)$. Since $P$ is abelian we have $P \subseteq C_G(P)$, whence $P \subseteq Z(N_G(P))$. We now can apply Burnside's Normal $p$-complement Theorem, which implies that $P$ has a normal complement $N$, that is $G=PN$ and $P \cap N=1$. Note that $|G/N|=p$.
Look at the image of $H$ in $G/N$. Then $G=HN$, or $HN=N$. In the latter case $H \subseteq N$, and $|G:H|=|G:N|=p$, whence $H=N$ and we are done if we can refute the first case. If $G=HN$, then $|G:H \cap N|=|G:N|\cdot|N:H \cap N|=|G:N|\cdot |G:H|=p \cdot p=p^2$, contradicting the fact that $|G| \mid |S_p|$. The proof is now complete.
Corollary 1 Let $G$ be a finite group and let $H$ be a subgroup with $|G:H|=p$, the smallest prime dividing the order of $G$. Then $G' \subseteq H$. In particular, $H$ is normal.
Corollary 2 Let $G$ be a finite group and let $H$ be a subgroup with $|G:H|=p$ and gcd$(|H|,p-1)=1$. Then $H$ is normal.
Observe that this last result renders a well-known result for $p=2$! Finally for fun:
Corollary 3 Let $G$ be a finite group of odd order and $H$ a subgroup with $|G:H|=65537$. Then $H$ is normal.