Number of automorphisms of a direct product of two cyclic $p$-groups Suppose I have $G = \mathbb{Z}_{p^m} \times \mathbb{Z}_{p^n}$ for $m, n$ distinct natural numbers and $p$ a prime. Is there a combinatorial way to determine the number of automorphisms of $G$?
 A: $\def\ZZ{\mathbb{Z}}$
It's worth pointing out that Arturo's second method generalizes very nicely to all abelian $p$-groups: Consider $G = \bigoplus \ZZ/p^{\lambda_i}$; let $\mu_{\ell}$ be the number of $\lambda$'s which are equal to $\ell$; let $r$ be the number of summands of $G$ (so $r = \sum \mu_i$).
Then $\mathrm{End}(G) \cong \bigoplus \mathrm{Hom}(\ZZ/p^{\lambda_i}, \ZZ/p^{\lambda_j}) \cong \bigoplus \ZZ/p^{\min(\lambda_i, \lambda_j)}$ as abelian groups.
An element of $\mathrm{End}(G)$ is invertible if and only if its image in $\mathrm{End}(G/pG)$ is invertible, by Nakayama's lemma.  Now, $G/pG \cong (\ZZ/p)^r$, so its endomorphism ring is $\mathrm{Mat}_{r \times r}(\ZZ/p)$. The map $\mathrm{End}(G) \to \mathrm{End}(G/pG)$ is NOT surjective. Rather, the image is the block-upper-triangular matrices, where the sizes of the blocks are the $\mu_i$. Such a matrix is invertible if and only if the diagonal blocks are invertible. So the fraction of $\mathrm{End}(G)$ which is made up of invertible elements is
$$\prod_{\ell} \frac{|\mathrm{GL}_{\mu_{\ell}}(\ZZ/p)|}{|\mathrm{Mat}_{\mu_{\ell} \times \mu_{\ell}}(\ZZ/p)|} = \prod_{\ell} \frac{(p^{\mu_\ell}-1)(p^{\mu_\ell}-p) \cdots (p^{\mu_\ell} - p^{\mu_\ell -1})}{p^{\mu_{\ell}^2}}$$
$$=\prod_{\ell} \left( 1- p^{-1} \right) \left( 1-p^{-2} \right) \cdots \left( 1-p^{-\mu_{\ell} +1} \right).$$
Putting it all together, $|\mathrm{Aut}(G)|$ is found by multiplying the above formula by $|\mathrm{End}(G)|$ (computed in the second paragraph), and we get
$$|\mathrm{Aut}(G)| = p^{\sum_{i,j} \min(\lambda_i, \lambda_j)} \prod_{\ell} \left( 1- p^{-1} \right) \left( 1-p^{-2} \right) \cdots \left( 1-p^{-\mu_{\ell} +1} \right).$$
A: I'm assuming that $p$ is a prime.
Assume $m\geq n$, let $x$ be a generator of the first cyclic group, and $y$ a generator of the second cyclic group.
Method 1. An endomorphism of $G$ is completely determined by the images of $x$ and $y$:
\begin{align*}
x &\mapsto x^a y^b\\\
y &\mapsto x^c y^d
\end{align*}
Since the image of $y$ must have order dividing $p^n$, the order of $x^c$ must divide $p^n$, so $c$ must be a multiple of $x^{p^{m-n}}$. So in fact, we can write
\begin{align*}
x &\mapsto x^a y^b\\\
y &\mapsto x^{cp^{m-n}} y^d
\end{align*}
$a$ is determined up to congruence modulo $p^m$; $b$ and $d$ are determined modulo $p^n$; and $c$ is determined up to congruence modulo $p^{n}$.
This can be "coded" with a matrix:
$$\left(\begin{array}{cc}
a & b\\
cp^{m-n} & d
\end{array}\right)$$
where the first column is taken modulo $p^m$ and the second column is taken modulo $p^n$. An easy computation shows that composition of these endomorphisms corresponds to matrix multiplication, and that matrix multiplication is well-defined (thanks to that factor of $p^{m-n}$ in the bottom left).
The matrix is invertible if and only if $ad - cdp^{m-n}$ is relatively prime to $p$. So the endomorphism is an automorphism if and only if $\gcd(ad - cdp^{m-n},p) = 1$. Note again that $a$ is determined modulo $p^m$, while $b$, $c$, and $d$ are determined modulo $p^n$.
If $m\gt n$, then the endomorphism is an automorphism if and only if both $a$ and $d$ are relatively prime to $p$, so the number of endomorphisms is $\phi(p^n)\phi(p^m)p^{2n}$: $\phi(p^m)$ choices for $a$, $\phi(p^n)$ choices for $d$, and arbitrary choices for $c$ and $d$ modulo $p^n$. This does not seem to match your computations.
If $m=n$, then you are simply looking for invertible matrices over $\mathbb{Z}_{p^m}$, and there are well-known formulas for that, but you say you are not interested, so I don't need to count them.
Method 2. Every map into $\mathbb{Z}_{p^m}\times\mathbb{Z}_{p^n}$ is determined by a pair of maps, one into $\mathbb{Z}_{p^m}$ and one into $\mathbb{Z}_{p^n}$. Every map from $\mathbb{Z}_{p^{m}}\times\mathbb{Z}_{p^n}$ is determined by a pair of maps, one from $\mathbb{Z}_{p^m}$ and one from $\mathbb{Z}_{p^n}$. So every endomorphism of $\mathbb{Z}_{p^m}\times\mathbb{Z}_{p^n}$ is determined by a $4$-tuple of maps, $(h_{mm},h_{mn},h_{nm},h_{nn})$, with $h_{ij}\colon\mathbb{Z}_{p^i}\to\mathbb{Z}_{p^j}$.
The number of possible $4$-tuples is then $(p^m)(p^n)^3 = p^{m+3n}$.
The map corresponding to the $4$-tuple $(h_{mm},h_{mn},h_{nm},h_{nn})$ is an automorphism if and only if $h_{mm}$ and $h_{nn}$ are both automorphisms (here we are using the fact that $m\gt n$; if $m=n$, then this claim does not hold). Thus, the number of choices is reduced: there are $\phi(p^m)$ choices for $h_{mm}$ and $\phi(p^n)$ choices for $h_{nn}$. However, $h_{mn}$ and $h_{nm}$ are free, each of them with $p^n$ choices. So the total number is $\phi(p^m)\phi(p^n)p^{2n}$, as before.
Summary. If $n\neq m$, the number of automorphisms is $\phi(p^m)\phi(p^n)p^{2\min(m,n)}$.
Coda. The general case of a direct product of finite groups is treated in

*

*J.N.S. Bidwell, M.J. Curran, and D.J. McCaughan, Automorphisms of direct products of finite groups, Arch. Math. (Basel) 86 (2006) no. 6, 481-489, MR 2241597 (2007e:20047)

(I had figured out the case of cyclic $p$-groups for a paper from scratch before I was given this reference). The main theorem is
Theorem. Let $H$ and $K$ be finite groups with no common direct factor. Then $\mathrm{Aut}(H\times K) \cong\mathcal{A}$, where $\mathcal{A}$ is the group of all $2\times 2$ matrices
$$\left(\begin{array}{cc}
\alpha & \beta\\
\gamma & \delta
\end{array}\right)$$
with $\alpha\in\mathrm{Aut}(H)$, $\delta\in\mathrm{Aut}(K)$, $\beta\in\mathrm{Hom}(H,Z(K))$, $\gamma\in\mathrm{Hom}(K,Z(H))$, and in particular
$$|\mathrm{Aut}(H,K)| = |\mathrm{Aut}(H)||\mathrm{Aut}(K)||\mathrm{Hom}(H,Z(K))||\mathrm{Hom}(K,Z(H))|.$$
