for prime $q\ge p$, integer $m\ge 0$, any group with order $p^2 q^m$ is not simple I tried the two ways, but both are all failed. First, counting the order of union of all Sylow $p$, $q$-subgroup. Second, group action from orginal group to set of cosets by Sylow subgroup.
Is there any more tools for proving problems like this?
 A: First, if $p=q$, then it is the $p$-group of $p^{m+2}$, in which $p$-group of $p^{m+1}$ is the normal subgroup. So assume $p<q$. Similarly assume $m>0$.
We prove that the given group $G$ must have a nontrivial normal subgroup. We consider following two cases.
(1). $m=1$
By Sylow theorem, the number of Sylow $q$-subgroup $n_q|1,\:p,\:p^2$, and $n_q\equiv1\mod q$. 


*

*If $n_q=1$, then there is only one Sylow $q$-subgroup which is normal. 

*If $n_q=p$, then since $p<q,\:n_q\not\equiv1\mod q$. So this is impossible.

*If $n_q=p^2$, then there are $p^2$ Sylow $q$-subgroup and at least $2$ Sylow $p$-subgroup (for otherwise Sylow $p$-subgroup would be normal). Let $H,K$ be any 2 distinct Sylow $q$-subgroups and $|H|=|K|=q$. Since $H\cap K<H$, $|H\cap K|=1$ or $q$.  If $|H\cap K|=q$, then $H=K$, which is impossible. So $|H\cap K|=1$. Similarly let $H,K$ be 2 distinct Sylow $p$-subgroups and $|H|=|K|=p^2$. Then $|H\cap K|=1,\:p$ or $p^2$.  If $|H\cap K|=p^2$, then $H=K$, which is impossible.  So $|H\cap K|\leqslant p$. So there are at least 
$$p^2(q-1)+2(p^2-1)-(p-1)=p^2q+p^2-p-1$$
elements in the group which is more than the given group with order of $p^2q$, which is impossible.
(2). $m>1$
Let $H,K$ be 2 distinct Sylow $q$-subgroups. Suppose $|H\cap K|<q^{m-1}$. Then 
$$
|HK|=\frac{|H||K|}{|H\cap K|}>q^{m+1}
$$
But since $|HK|\leqslant |G|=p^2q^{m}$, this is impossible. So $|H\cap K|=q^{m-1}$.
By Normal subgroup of prime index, since $q$ is the smallest prime index dividing $q^m$ 
$$
H\cap K\vartriangleleft H \quad \text{and}\quad H\cap K\vartriangleleft K
$$
Thus 
$$
H\subset N_G(H\cap K) \quad \text{and}\quad K\subset N_G(H\cap K) 
$$
where $N_G(H\cap K)$ is the normalizer of $H\cap K$ in $G$. So
$$
|N_G(H\cap K)|\geqslant |HK|=\frac{|H||K|}{|H\cap K|}\geqslant q^{m+1}
$$
Note that $HK$ may not be a subgroup but $N_G(H\cap K)$ is. 
$|N_G(H\cap K)|=q^{m+1}$ is impossible since $N_G(H\cap K)<G$ and $|N_G(H\cap K)|\nmid|G|$. 
If $|N_G(H\cap K)|>q^{m+1}$, then $|N_G(H\cap K)|\geqslant pq^{m+1}$. But
$$
|N_G(H\cap K)|>p^2q^{m}=|G|
$$
which is impossible. So $H\cap K\vartriangleleft G$.
A: The cases where $m=0$ or $q=p$ are already discussed in the other post. So suppose $q>p$ and $m\geq 1$.
If $m=1$: Then any two $q$-Sylow subgroups must intersect trivially, so if $n_q = p^2$, then there must be $p^2(q-1) + 1$ elements in the group. Also, $n_q \neq p$ since $p\neq 1\pmod{q}$. So $n_q = 1$ the $q$-Sylow subgroup is normal.
If $m>1$: Suppose $P_1$ and $P_2$ are two distinct $q$-Sylow subgroups, then set $H=P_1\cap P_2$, then
$$
p^2q^m \geq |P_1P_2| = |P_1||P_2|/|H| = q^{2m}/|H| \Rightarrow |H| \geq \frac{q^m}{p^2} > q^{m-2}
$$
Since $|H| \mid q^m$ and $|H| < q^m$, it follows that $|H| = q^{m-1}$, and so $H$ is normal in both $P_1$ and $P_2$. Furthermore,
$$
|P_1P_2| = q^{2m}/q^{m-1} = q^{m+1}
$$
Now, $P_1$ and $P_2$ are both contained in $N_G(H)$, the normalizer of $H$ in $G$. Hence,
$$
P_1P_2 \subset N_G(H) \Rightarrow |N_G(H)| \geq q^{m+1}
$$
But $|N_G(H)| \mid p^2q^m$, so $|N_G(H)| = p^2q^m$, and so $H \vartriangleleft G$.
