The definition of End(A).

My professor defined $End(A)$ as follows:

"If $A$ is an abelian group, then $End(A):=\{ h: h:A \to A$ is a group homomorphism $\}$"

While the book defined it as: "$End(A) := Hom_{_R}(A,A)$",

So I am asking $End(A)$ is the group homomorphism or the $R$-module homomorphism?

Could anyone explain this for me please?

All abelian groups are $\mathbb{Z}$-modules, so take $R=\mathbb{Z}$ and they agree. Your professor just gave a slightly less general definition only in terms of abelian groups.
The notation $End(A)$ has different meanings in different contexts. It generally means the set (or abelian group, or module, or...) of all homomorphisms $A\to A$. What "homomorphism" means (group homomorphisms, $R$-module homomorphisms, ...) depends on context: generally, it's homomorphisms with respect to whatever structure you care about on $A$ at the moment. To disambiguate between group homomorphisms and module homomorphisms, the $R$-module homomorphisms are sometimes written as $End_R(A)$ instead of just $End(A)$.