Actually, Peano Arithmetic does not uniquely determine $\mathbb{N}$, even if you fix the cardinality of the structure you want (also called a model). These are the so-called "non-standard" models of PA, and there are a lot of them: $2^{\aleph_0}$ to be more precise. They all look like "end-extensions" of $\mathbb{N}$. That is, $\mathbb{N}$ plus a bunch of stuff at the end. There are some things you can say about what such a model looks like and how its arithmetic behaves. Read more about that here.
That link also shows how to show that such models exist, but let me outline a construction here using Godel's incompleteness theorem. Godel proved, among other things, that there is a logical sentence $G$ that is true in $\mathbb{N}$, but cannot be proved or disproved from the axioms of PA. Thus if we consider the axioms PA $+ \neg G$, Peano Arithmetic plus the the negation of $G$, this is consistent, and hence has a model by standard results in logic. Of course, any model of PA $+ \neg G$ also satisfies PA, but this model disagrees with $\mathbb{N}$ on the truth of $G$, so they cannot be isomorphic.
To go into a little more detail about your question as a whole, let me explain what Peter Smith brought up in the comments. There is a notion of a "countably categorical" or "$\omega$ - categorical" theory. This is a collection of axioms where every countable model is isomorphic. You can think of this as uniquely determining a (countable) structure. Given a collection of axioms (which we will call a theory), it is of interest to determine when it is or is not $\omega$-categorial, how many models there are, and perhaps how much "variation" there can be. There are many equivalent conditions for $\omega$-categoricity outlined here, but I will try to give you some context before sending you to read Wikipedia.
You can think of a theory as the basic operating system behind your model. The theory tells you what axioms you must satisfy. However, there are all sorts of possible extra software packages you can install. These extra packages are called "types". Some of these types are not compatible with your operating system, so you give up on them immediately. Some software packages are compatible with your operating system, but not with each other. Each "type" promises you the existence of a collection of special objects in your model. A $1$-type gives you one special element, whereas a general $n$-type gives you a collection of $n$ elements satisfying certain properties with respect to each other, and the model in general. Unless I'm mistaken, I don't think that types are the final word on the issue of non-isomorphic models, but they are certainly important to that end. Understanding what sort of types are compatible with our theory, and in what combination, tells us a lot about the sorts of models our theory can have.