How can one prove that there are no subgroups of $S_5$ with order 40?
Every subgroup of $S_n$ is either all even of half-even and half-odd.
Suppose $H$ is a subgroup of order 40 in $S_5$.
Case 1: $H$ is all even. Then $A_5$ has a subgroup of order 40. This is absurd since 40 does not divide $|A_5|=60$.
Case 2: $H$ is half-even and half-odd. Thus $K = H \cap A_5$ is a subgroup of $A_5$ of order $40/2=20$. Consider $A_5$ acting on the left cosets of $K$: $\sigma \cdot \tau K = (\sigma\tau)K$. Thus we have $A_5$ acting non-trivially on a set of 3 elements (the index of $K$ in $A_5$ is $60/20=3$). But $A_5$ is simple so any non-trivial action is faithful -- that is -- the corresponding permutation representation $\varphi:A_5 \to S_3$ is injective. But this is absurd since $|A_5|=60 > |S_3|=6$.
Therefore, no such subgroup can exist.
If such a subgroup $H$ exists, then the normal core $N$ of $H$ in $S_5$ has order 20 or 40. In both cases a Sylow 5-subgroup $P$ of $N$ is characteristic in $N$. Therefore $P$ is normal in $S_5$. However, $\langle (12345)\rangle$ and $\langle (12354)\rangle$ are two distinct Sylow 5-subgroups of $S_5$, a contradiction.
In general, a group of order 120 having two distinct Sylow 5-subgroups cannot have a subgroup of order 40.