How is $\mathbb N$ actually defined? I know perfectly well the Peano axioms, but if they were sufficient for defining $\mathbb N$, there would be no controversy whether $0$ is a member of $\mathbb N$ or not because $\mathbb N$ is isomorphic to a host of sub- and supersets of $\mathbb Z$.
So it seems that mathematicians, when specifying $\mathbb N$, rarely mean the set of natural numbers according to its definition but rather some particular embedding into sets defined by different axiomatic systems.
Doesn't this sort of misstate then the role of the Peano axioms as the defining characteristics of $\mathbb N$?  It seems like most of the time the Peano axioms are not more than a retrofit to a subset of $\mathbb Z$ or other sets defined by a different axiomatic system with one or several axioms incompatible with the Peano axioms.
 A: Well, usually they do mean "the set," just different mathematicians mean different sets. It's like a language difference. There are lots of cases in math of this sort. For example, "the standard normal distribution" means something different to different mathematicians. Basically, good luck trying to sort this out.
Historically, the natural numbers were essentially the ordinals. So, if you could say "I came in $n$th place in a race" then $n$ was a natural number. 
The distinction between ordinal number and cardinal (counting) number is essentially the logical error that led to problems with the number zero. When you count a pile of beans like a child would, you are assigning an ordinal to each bean. "This is the first bean, this is the second bean, etc." The last ordinal used is then treated like a cardinal number. 
(Aside: It is actually not 100% intuitive that this counting technique works - that no matter how we assign the ordinals to a finite set, we get the same last ordinal. We need to be taught that this works. There is a Sesame Street video that counts kids on the screen, then has the kids run around into a different configuration, and counts them again - essentially assigning different ordinals to each kid each time, demonstrating that the final ordinal is always the same, no matter how you assign them. This counting technique breaks when trying to count infinite sets.)
Since there was no ordinal $0$, the difficulty with "counting" empty sets was that there was no last bean in the process. So they'd simply say, "there are no beans," rather than giving a count of zero.
Set theory starts ordinals at $0$, instead, for a variety of reasons. This means that, in a race, the person who places $n$th was beaten by $n$ people. So $0$th place would be beaten by zero people, $1$st place would be beaten by one person, etc. Nobody but mathematicians treats zero as an ordinal, but it does have its advantages. In set theory, the $n$th ordinal is a set of the previous ordinals - you can see that as "my race result is the set of results of the people who beat me."
So in set theory: $\emptyset$ is an ordinal, and if $x$ is an ordinal, then $x\cup\{x\}$ is the "next ordinal" - the person in the race with the result directly after me was beaten by me and everybody who beat me. Set theory assumes there exists a set closed under this operation $x\mapsto x\cup\{x\}$, and then shows there is a unique minimal such set, which is called the natural numbers - minimality essentially being equivalent to induction.
(Then we get into infinite cardinals and ordinals, where the relationship becomes way more complicated. Race results, for example, is really only a good metaphor in the finite case.)
A: The original Peano's axioms (1889, page 1) started from $1$ :

Axiomata

$1 \in N$ [...].


This "tradition" is followed by many mathematicians, like :

*

*Edmund Landau, Foundations of analysis : The arithmetic of whole rational irrational and complex numbers (ed or 1930, page 2) :


Axiom 1 : $1$ is a natural number.

In the mathematical logic tradition, following G.Frege (see e.g. The Foundations of Arithmetic (1884), transl J.L.Austin, page 90) and B.Russell (see Principles of Mathematics (1903), §120. Peano’s indefinables and primitive
propositions) is common to "start from" $0$.
See : Kurt Gödel (1931). "Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme, I". Monatshefte für Mathematik 38, page 177 :

The primitive signs of the system $P$ are the following:

I. Constants: "$\sim$" (not), "$\lor$" (or), "$\Pi$" (for all), "$0$" (zero), "$f$" (the successor of), "(", ")" (parentheses); [...]

The following formulas (I-V) are called axioms [...] :

I.1. $\sim (fx_1 = 0 )$,

[...].


You can see also :

*

*Paul Benacerraf, What Numbers Could Not Be (1965), The Philosophical Review, 74.

A: I don't think that Peano axioms are sufficient for defining $\mathbb{N}$. For one thing, Peano axioms form a theory, not a particular structure. Additionally, the theory of Peano arithmetics is not complete, so it even doesn't define a structure up to elementary equivalence.
On the other hand, there is a standard model of Peano arithmetics in the standard set theory (ZFC), namely the smallest infinite ordinal – as described in other answers.
Also, you can easily construct a structure that is different but isomorphic to $\mathbb{N}$. But that is trivial and matematitians often think of structures “up to isomorphism”. So they often do not make difference for example between zero as a natural number and zero as an integer, even though they are strictly speaking different objects (different sets) according to the standard set-theoretic formal construction.
A: How is a chair defined? Surely something as simple and fundamental as a chair should have a reasonable definition.
In the dining room, the table is certainly not a chair. Because you don't eat from a chair. But in the university lobby, sometimes I would sit on the table when talking to someone who's standing up. And maybe I decided to sit on the floor and eat off the chair (with plates, mind you), because I needed to hide from someone.
So what is a chair? Does the number of legs define it? Does its location in the house define it? Does anything else?
Well. The honest truth is that a chair is an abstract idea, and it is not universally and globally well-defined. At some times a chair is something like this, and at other times it could be a completely different thing. The important thing is that within the context of the conversation, if I say "chair" it will be clear what I meant by that term.
And now we finally approach the point I am trying to make. $\Bbb N$ is an abstract object. When I talk to a fellow set theorist, it is clear that when I say $\Bbb N$, I mean the set of finite von Neumann ordinals (often called $\omega$). When I talk to someone who's doing a lot of category theory it is clear, at least to him I guess, that $\Bbb N$ is the initial object in the category of free monoids. When I talk to someone whose main interest is analysis, she might have a clear and intuitive as what are the real numbers, and from that she will derive the meaning of $\Bbb N$ as the smallest inductive set.
Sometimes it will be beneficial and useful to include $0$ as a natural number. For example if a set is finite if and only if its cardinality is a natural number, then surely $0=|\varnothing|$ is a natural number.
Sometimes it will be cumbersome to include $0$, when I was to talk often about sequences that look like $\frac1n$ having to add "for $n>0$" everywhere is a terrible crime against yourself, the reader and the listener. So just excluding $0$ works better.
Since you brought up the Peano arithmetic axioms, it is perfectly reasonable to develop those in a language which include $0$, and in a language which does not include $0$ (in which case $1$ is the minimal element, and you need to modify a few axioms accordingly).
So what is $\Bbb N$? It is an abstract idea, which you can implement, or realize in many various ways. Some are more concrete, some are less concrete. But it is an abstract object, that you know satisfy some properties, and perhaps the most important one is that it has a linear ordering that (1) has no maximal element; and (2) every proper initial segment is finite. 
Why am I choosing the order here rather than the arithmetic structure? Because from this order we can define (in the broad sense of the word) the arithmetic structure, and everything else. With or without $0$. And this order gives us a uniqueness property. Every two linear orders with these two properties are isomorphic (and the isomorphism is unique, too).
So it really doesn't matter what is the set $\Bbb N$, because there are way too many options for interpreting $\Bbb N$ as a concrete set. It matters that we can use induction on that set, and that we know there is at least one set like that.
A: In the set theory course I took, natural numbers were not defined by Peano's axioms. One can define an inductive set in this manner: $I$ is inductive set if
$$\emptyset\in I\ \wedge\ \forall x\in I\  ((x\cup \{x\})\in I) $$
We say that $n$ is natural number if $n\in I$ for any inductive set $I$. Some examples of natural numbers are:
\begin{align}
0 &= \emptyset\\
1 &= \{\emptyset\}\\
2 &= \{\emptyset,\{\emptyset\}\}\\
&\ \ \vdots\\
n &= \{0,1,\ldots ,n-1\}
\end{align}
But, we still don't know if the set of all natural numbers exists. This is guaranteed by Axiom of infinity:

There exists an inductive set.

and with this we can prove existence of set $\mathbb N = \{n\ |\ n\ \text{is natural}\}$. Also, we can see that $\mathbb N$ is the smallest inductive set in the sense of set inclusion.
A: You have to start somewhere.
Some treatments of mathematics start with Peano's axioms for the natural numbers, and use that to define other number systems which extend the natural numbers --- first integers, then rational numbers, real numbers, complex numbers,...
But that's not the only choice. You could equally well decide to start with the axioms for the real numbers, and use that to define other number systems: first natural numbers, integers, and rationals within the real numbers, then the extension to complex numbers …
The above two methods can be thought of as dependent on "naive set theory". Because of that, perhaps a more theoretically satisfying place to start is with axioms for set theory such as ZF (the Zermelo Frankel axioms) or ZFC (the Zermelo Frankel axioms with the axiom of Choice tacked on). In these treatments, the natural numbers are usually defined as the set of finite ordinals (the existence of which is one of the ZF axioms, and the Peano axioms must then be proved), in which case the number zero, which is defined as the empty set, would be included. Alternatively, the natural numbers could be defined as the finite nonempty ordinals, in which case zero would not be included.
As you can see, there's different choices of where to start, and different people may make different choices. 
