Why are there so many groups (up to isomorphism) of order $p^n$ for $n>2$, especially when compared to groups of similar sized order?

While bounds on the number of isomorphism classes of groups of order $p^n$ where $p$ is prime have been known for quite a while (such as the work of Higman$^{}$ and Sims$^{}$) which give us the bounds for $f(n,p)$ (which returns the number of isomorphism classes of groups of order $p^n$) as follows:

$f(n,p) = p^{An^3} \operatorname{where} A(n,p) = \frac{2}{27} + O(n^{-1/3})$

However, while this tells us that for a fixed $p$ the value of $f(n,p)$ grows very rapidly with n, when I've been reading through the papers I don't think I have enough knowledge to understand why exactly they should grow this rapidly - for example, the number of abelian groups of order $p^n$ is simply the number of partitions of $n$, and while this number grows quite rapidly with n, this number is tiny in comparinson to the number of non-abelian groups of the same exponent.

Therefore, would it be possible for someone to try and explain to me some reasons for why there are so many groups of order $p^n$ for $n>2$ (as it is fairly straightforward to categorize such groups for $n=1$ and $2$), and also why there are so many such $p$-groups when compared to groups of similar sized order? (I suspect these two questions are closely linked in some respects, which is why I ask them both, although answers on one or the other are equally appreciated)

For those who would like them, the references to the papers of Higman and Sims are below:

$$ - Proc. London Math. Soc. (1960) s3-10 (1): 24-30. doi: 10.1112/plms/s3-10.1.24

$$ - Proc. London Math. Soc. (1965) s3-15 (1): 151-166. doi: 10.1112/plms/s3-15.1.151

• I am not sure that this is a plausible explanation, but there may be some correlation with the fact that the Lagrange Theorem has an inverse for $p$-groups. – Andrea Mori Jul 31 '13 at 14:12
• One possible reason is that the automorphism group of a $p$-group will generally have order divisible by a fairly large power of $p$, which means there is a large number of possible semidirect products to be made when increasing the order. And then there are the possibilities of non-split extensions of course. – Tobias Kildetoft Jul 31 '13 at 14:13
• I think one reason would be that the number of generators of a cyclic group of order $p^r$ is $p^{r-1}(p-1)$ which is divisible by $p^{r-1}$ - so the automorphism group will give plenty of scope for nonabelian group extensions (eg semidirect products). – Mark Bennet Jul 31 '13 at 14:16
• @AndreaMori Are you referring to the Sylow theorems? – Andrew D Jul 31 '13 at 14:17
• As to why there are so many $p$-groups, just look at the paper of Higman and see how they are constructed. It is harder to explain why $p$-grpoups predominate over all groups, and it has not been proved to be true, so it may not be! – Derek Holt Jul 31 '13 at 14:25

I don't think that a precise answer to your question exists (up to this moment). More or less your question is Question 2 in Mann's paper Some Questions about $p$-Groups (you can get it freely on the net), I find it better to refer you to Sections 1 and 2 of that paper, than any word I can say.