The morally correct definition of prime number is given by Euclid's lemma. If you have a ring that is an integral domain ($ab=0\implies a=0$ or $b=0$), that is, a set with sum, multiplication, all the known rules and a $0$ and a $1$, a non-unit non-zero element is said to be prime if $p\mid ab\implies p\mid a$ or $p\mid b$. Where $p\mid a$ means that $a=pq$ for some other $q$. If a number $p$ has this property and if $u$ is invertible, i.e. there is $v$ for which $uv=vu=1$, then $up$ has this property too. If for two numbers $a,b$ there is a unit $u$ for which $a=ub$, we say that $a$ and $b$ are associates. When we want to look at factorization of numbers, we thus take from the set of all primes of your domain, a set of representatives: that is, a subset of the primes such that every prime is associate to one of the primes in our representatives set, and such that no representatives are associates. In the domain $\Bbb Z$ of integers, the (positive) prime numers $2,3,\ldots$ are a set of representatives of all the primes of $\Bbb Z$, $\pm 2,\pm 3,\ldots$. The units of $\Bbb Z$ are $1,-1$, which is what you observed.