Suppose $G$ is a finite non-abelian group. Is it true that if $32 \nmid |G|$, then $G$ has at least one abelian centralizer?

• Did you mean order $96$? Did you check the structure of the examples, as it seems probable that the "extra" ones would just be direct products of small groups with the example of order $32$. – Tobias Kildetoft Oct 18 '13 at 7:51
As Derek Holt suggests in the comments, an extraspecial group of order $243 = 3^5$ is a counterexample. More generally, for any odd prime $p$, an extraspecial group of order $p^{2n+1}$ ($n \geq 2$) is a counterexample to your statement.
Proof: Let $|G| = p^{2n+1}$ be extraspecial, so $Z(G) = G'$ has order $p$. If $x \in Z(G)$, then $C_G(x) = G$ is nonabelian. If $x \not\in Z(G)$, the centralizer $C_G(x)$ has index $p$, since every conjugate of $x$ is contained in $xG'$. Suppose that $C_G(x)$ is abelian. Now for $y \not\in C_G(x)$, the intersection $C_G(x) \cap C_G(y)$ is central of index $p^2$, which implies $n = 1$ since $Z(G)$ has order $p$.