When is $f(x)=x^2$ an automorphism of a finite group G? I tried a few examples and found that it is an automorphism of $A_3$
Also, to satisfy the homomorphism property, $f(x)f(y)=f(xy)$, it must be true that $x^2y^2=(xy)^2$. This is true in abelian groups. However, I'm not sure how to continue.
 A: $f(x) = x^2$ is an automorphism of $G$ if and only if $G$ is abelian and no element in $G$ has order 2.
Proof: Considering $f(x) = x^2$, then $f$ is a homomorphism if and only if$\;\;\forall x, y \in G$ we have $f(xy) = (xy)^2 = f(x)(y) = x^2y^2$ which hold if  and only if $xy = yx$. Thus, the map is a homomorphism if and only if $G$ is abelian.
Now, such a homomorphism is an automorphism if and only if it's injective which holds if and only if its kernel $K = \{1\}$. This implies that $\forall x\in G$, if $x\neq 1$ then $x^2 \neq 1$. This ends the proof.
Note that no element having order $2$ is equivalent to all elements of $G$ having odd order. That's because if $|x| = 2k$ for some $k\in \mathbb{N}$, then $|x^k| = 2$.
A: The order of G should be odd.
Since if the order of G is even, there exist an element x whose square is equal to 1.
But the value of 1 is 1. So the function is not automorphism anymore.
So, you should add that G has odd order.
A: From $xxyy=x^2y^2=(xy)^2=xyxy$ it follows directly that $xy=yx$. That tells you that it is necessary for the group to be abelian if $f$ is indeed a grouphomomorphism. It is  directly clear that $f$ is a grouphomomorphism if the group is abelian. 
Shortly a group $G$ is abelian if and only if the map $x\mapsto x^{2}$ is a grouphomorphism.
A: I would like to add my proof, to give another view using conjugacy and the definition of a group center:
For the forward direction ($x \mapsto x^2$ being an automorphism $\implies$ $G$ is abelian with no element of even order), one can see that the image of $G$ under $\phi(x) = x^2$ is equal to the center of $G$. This stems from the fact that, for some $g \in G$, $xx = g$ and $x = gx^{-1}$, so $xgx^{-1} = x$ and more explicitly $x\phi(x)x^{-1} = \phi(x)$. Because $\phi$ is an automorphism $\phi[G] = G = Z(G)$, so $G$ is abelian. Likewise, because $\phi$ is an isomorphism, it has trivial kernel $Ker(\phi) = \{e\}$, so no non-identity element of $G$ can map to $e$ under $\phi$. But if there were some element of $G$ with even order, $x^{2n} = e$, then $x^{2n} = (x^2)^n = g^n = e$ which is a contradiction.
For the backward direction ($G$ is a finite abelian group with all elements of odd order $\implies$ $x \mapsto x^2$ is an automorphism ) it is clear from the other comments that $\phi$ is a homomorphism. We just need to show that $\phi$ is bijective and that its image is $G$. $\phi$ is surjective because for any $g \in G \ \exists x \in G \ni x^2 = g$ by definition of a group. $\phi$ is injective because $x^2 = e$ implies that $|x|$ is even, which is a contradiction, so $x = e$ necessarily, and the kernel is trivial. Again, by definition of a group, the range of $\phi$ is $G$ itself, and thus $\phi$ is an automorphism.
