What does it mean for an automorphism to centralize factor group $G/M$? 
Let $G$ be a group and $M$ be a normal subgroup of it. An automorphism $\phi$ centralizes the factor group $G/M$. 

What does it mean for an automorphism to centralize a factor group $G/M$?
I encountered the term here [http://arxiv.org/pdf/0803.4081].  In second paragraph he denote it by $\operatorname{Aut}^N (G)$.
 A: Given the paper mentioned by the OP and one of its references (with the exact same title—ugh!)...
Given $\phi\in\operatorname{Aut}(G)$ we say that $\phi$ centralizes a subgroup $H\subseteq G$ if $\phi(h)=h$ for all $h\in H$.  So $\phi$ fixes $H$ elementwise.  Equivalently, the restriction map gives a group homomorphism $\operatorname{Aut}(G)\to\operatorname{Aut}(H)$, and the automorphisms that centralize $H$ are exactly the kernel of this homomomorphism.
On the other hand, we say that $\phi\in\operatorname{Aut}(G)$ centralizes the quotient $G/M$ if and only if $gM=\phi(g)M$ for all $g\in G$.  Equivalently, $\phi(g)g^{-1}\in M$ for all $g\in G$.  Note, however, that the map $gM\mapsto \phi(g)M$ need not be a group homomorphism for arbitrary $\phi,M$, for this happens if and only if $\phi(M)\subseteq M.$  In particular, this map gives a group homomorphism for all $\phi\in\operatorname{Aut}(G)$ if and only if $M$ is characteristic in $G$ (by definition).  However, those $\phi\in\operatorname{Aut}(G)$ with $\phi(M)\subseteq M$ form a subgroup $T_M$ of $\operatorname{Aut}(G)$, and we have a well-defined group homomorphism $T_M\to \operatorname{Aut}(G/M)$ given by $\phi\mapsto( gM\mapsto \phi(g)M)$, and the automorphisms of $G$ that centralize $G/M$ are precisely the kernel of this homomorphism.  As noted before $T_M=\operatorname{Aut}(G)$ if and only if $M$ is characteristic in $G.$
For example, in the notation of the OP's reference $$\operatorname{Aut}^{Z(G)}(G) = \{ \phi\in\operatorname{Aut}(G) \ | \ \phi(g)g^{-1}\in Z(G) \ \forall g\in G\}$$
is the central automorphism group, which is precisely the centralizer subgroup $C_{\operatorname{Aut}(G)}(\operatorname{Inn}(G))$, and
$$ \operatorname{Aut}_{Z(G)}(G) = \{ \phi\in\operatorname{Aut}(G) \ | \ \phi(z)=z \ \forall z\in Z(G) \}$$
are the automorphisms of $G$ that fix $Z(G)$ elementwise, and finally
$$ \operatorname{Aut}_{Z(G)}^{Z(G)}(G) = \operatorname{Aut}_{Z(G)}(G) \cap \operatorname{Aut}^{Z(G)}(G)$$
are the central automorphisms that fix the center elementwise.
