Invariant factors of a Hadamard matrix divide the dimensionality of the matrix

I want to show that if $H$ is a Hadamard matrix of size $N$ then every invariant factor of $H$ divides $N$. My attempt at solution was noting that if $HH^{\intercal}=NI$ then $H \frac{1}{N}H^{\intercal}=I$ so $H$ has inverse $1/N$ times its transpose. What I guess I wanted to show was something like, if $H$ and $H^{-1}$ could be put into Smith Normal Form, $H^{-1}$'s diagonals would be the same as the entries in $H$, but divided by $N$, so every term in $H$ needs to have a factor of $N$. Does that make sense, and is that a sufficient argument?

-
Let $d_1\mid\cdots\mid d_N$ be the invariant factors of $H$. Then the matrix $H^T=NH^{-1}$ has the same invariant factors. On the other side the invariant factors of the integral matrix $NH^{-1}$ are $\frac{N}{d_N}\mid\cdots\mid \frac{N}{d_1}$, that is $d_id_{N-i+1}=N$ for all $i$.
Makes sense and straightforward. It helped to see it when I tried with a few actual $H$'s. –  tacos_tacos_tacos Nov 20 '12 at 16:08