Theorem. Let $G$ be a finite, non-abelian $p$-group all of whose proper subgroups are abelian. Then $|G'|=p$.
Take a counterexample of minimal order. Assume that exist a $H$ such that $1<H<G'$.
Then (by $G'\leq \Phi (G) \leq Z(G)$) $H\vartriangleleft G$. From this we deduce we can assume $|G'|\leq p^2$.
Then? How am I supposed to continue?
$G'$ is elementary abelian since $G$ is Frattini-in-center.