For abelian groups, the ring End(A) is very important. As far as non-abelian groups A go, End(A) is not even (usually considered) a group.
"Adding" homomorphisms doesn't work in the non-abelian case.
If you define (f+g)(x) = f(x) + g(x), then (f+g)(x+y) = f(x+y) + g(x+y) = f(x) + f(y) + g(x) + g(y), but (f+g)(x) + (f+g)(y) = f(x) + g(x) + f(y) + g(y). To conclude that:
f(y) + g(x) = g(x) + f(y)
are equal, you use that + is commutative, that A is abelian. More precisely, if you take f=g to be the identity endomorphism, then f+g is an endomorphism iff A is abelian.
"Composing" homomorphisms doesn't work to form a group, since they are not invertible.
Aut(A), the group of invertible endomorphisms, does form a group. Aut(A) does not determine if a group is abelian or not: 4×2 and the dihedral group of order 8 have isomorphic automorphism groups.
Instead of a ring, End(A) sits inside the "near-ring" of self-maps. See the wikipedia article on nearring for an explanation.