Difference between a number and a set with one number I am studying Analysis in Tao's textbook and he mentions the following: 

The set $\{3, \{3, 4\}, 4\}$ is a set of three distinct elements, one of which happens to itself be a set of two elements. However, not all objects are sets; for instance, we typically do not consider a natural number such as $3$ to be a set. (The more accurate statement is that natural numbers can be the cardinalities of sets, rather than necessarily being sets themselves)

but my question is wouldn't $\{3\}$ be a set of cardinality one? Is this just a matter of notation (if we include the curly braces then we consider it a set and if not then it's just a natural number)?
 A: Set's are collections of objects.  The objects that can be collected in the sets can be sets themselves.
This is a conceptual block for many students.  If a set is a collection and it has collections within it, then, they ask, aren't the items within subcollections also be in the collection? And the answer to that is a resounding and absolute !!!!!NO!!!!!!.  And the reason is a simple basic no concept, no interpretion, no computation.  A set is a list of objects, with NO interpretation.  If a set has:  a mermaid, a bag of dog food, and the English alphabet, then the set has three things in it.  1) a mermaid, 2) a bag of dog food, and 3) the English alphabet.  It does not have several pieces of kibble; it does not have A, K, Q, and P.  It does not have "slightly venomous saline saliva" and "a respiratory system that functions under water as well as in the air"-- even though those are things contained in the objects, they are not objects in the set itself.
Perhaps it be easier to think a set as a "list".  If  you have a list of three items:  Babar the Elephant, Roosevelt Franklin Elementary School, and the starting lineup of the 1942 New York Yankees; then the list has three things on it.  The components of the items in the list are not themselves on the list.  BUt then...list implies things that aren't necessary; order doesn't matter, definition doesn't matter.  A set is basically dumping things in a bag and carrying them.
But I think you get all that.
To answer your questions:

"wouldn't {3} be a set of cardinality one?"

Indeed, yes.

Is this just a matter of notation (if we include the curly braces then we consider it a set and if not then it's just a natural number)?

Pretty much.
....
I think Tao's main point is:

The more accurate statement is that natural numbers can be the cardinalities of sets, rather than necessarily being sets themselves

There is an issue what is a natural number.  Mathematics is bootstrapping from ... as little as we can get away with and what are we doing when we count?
Well, It may be that he is going to get into the set constructionist idea of constructing the natural numbers from sets.
$0$ is the empty set.
$1$ is the set containing the empty set.
If you have defined $n$ as as certain set then $n+1$ will be set that is the union that set and the set containing that set.
So $0 = \emptyset$
$1 = \emptyset \cup \{\emptyset\}=\{\emptyset\}=\{0\}$
$2 = 1 \cup \{1\} = \{\emptyset\} \cup \{\{\emptyset\}\} = \{\emptyset, \{\emptyset\}\}=\{0,1\}$.
$3 = 2\cup \{2\} = \{\emptyset, \{\emptyset\}\}\cup \{ \{\emptyset, \{\emptyset\}\}  \}= \{ \emptyset, \{\emptyset\}\},\{\emptyset, \{\emptyset\}\}  \}=\{0,1,2\}$
and so on.
And much as I've come to value and rely on this concept of natural numbers, I sincerely hope your reaction is "What the #@&! are you talking about!"
Natural numbers aren't sets.  Sets are sets.   Right?  
But what are natural numbers?  Or for that matter what is ... anything.  The set theorist view is everything is a concept of sets.  We start with the idea of set with nothing in it, and the idea of putting things into sets and we can constructing a succession of objects, (and "succession" is equivalent to counting things one by one) by putting the previous object into itself to make the next object.
Whew!......
But as Tao points out... the natural numbers aren't actually the sets that we make this way, but the concept of the cardinality of how many elements in the sets we create.
Our first set is $K_0 = \emptyset = \{\}$ a set with nothing in it.  And our first natural number is $0 = |K_0|$ the number of elements in it.  $K_0$ has no elements in it.
Our next set is $K_1 = \{K_0\}$ a set with one set in it. And our next natural number is $1 = |K_1|$ the number of elements  in it.
Our next set is $K_2 = K_1 \cup \{K_1\} = \{K_0, K_1\}$ and $2 = |K_2|$ which has two elements in it.
And $K_3 = \{K_0,K_1,K_2\}$ and $3=|K_3|$.
And so on.
SO I suspect that Tao is trying to get to two birds:
1) A set can have sets as elements, but the elements within those sets are not elements of the "upper" set.
3) Natural numbers can be elements in sets, but not sets themselves, but can represent the cardinality of sets.
So $|\{3\}| = 1$.  But $|\{0,1,2\}| = 3$ and $|\{0,1,\{0,1\}\}|= 3$ and $|\{\emptyset, \{\emptyset\},\{\emptyset, \{\emptyset\}\}\}| = 3$.
This leads to the, initial apparently circular but actually valid, definition:  The natural number $n$ is defined of as the cardinality of the set $\{0,1,2,3,.....,n-1\}$.
(Sometimes it's a good idea for a mathematician to take a day off and go to the zoo and look at the elephants and then start fresh the next day....)
A: $\{ 3 \}$ is indeed a set of cardinality $1$. Whether $3$ is a set depends on your definition of the natural numbers. In some systems they are sets. I don't know what definition Tao uses.
A: As Ethan said, $\{3\}$ is a set of cardinality $1$. What Tao means in his example is that in the set $\{3, \{3, 4\}, 4\}$, the element $\{3, 4\}$ is a set but $3$ is not.
Maybe the confusion arises from his last statement:

Natural numbers can be the cardinalities of sets, rather than necessarily being sets themselves.

So for example, $3$ is the cardinaily of $\{3, \{3, 4\}, 4\}$, but he doesn't regard $3$ itself as a set. A thing to note as pointed in JMoravitz's comment and in Ethan's answer is that you can define the natural numbers to be sets; for example one can define $0$ to be $\emptyset$ ,  $1$ to be$\{\emptyset\}$, $2$ to be  $\{\emptyset, \{\emptyset\}\}$ and so on, but of course this is not the only way to do this!
