Why is the simplest set satisfying The Axiom of Infinity the von Neumann ordinals? The Axiom of infinity is colloquially defined as:

There exists a set X having infinitely many members

(see Wikipedia)
In the language of first-order logic, it's convention that the following statement is the simplest solution to the axiom.
$$\exists X[\emptyset\in X \wedge\forall y(y\in X \implies (y\cup\{y\})\in X)]$$
Notice that I've swapped out $S(y)$ for $y\cup\{y\}$ as $S(y):=y\cup\{y\}$
One fellow mathematician, bof, in the comments of the answer to this question, asks why isnt the following statement a more simple solution?
$$\exists X[\emptyset\in X \wedge\forall y(y\in X \implies \{y\}\in X)]$$
To which, Asaf Karagila, responds:

No, because the language is more complicated. 

Of course, not to discredit Karagila, I am assuming there is a more rigorous argument than the complication of the language. Regardless, bof's solution seems less complicated as it only withholds the operation of the union, while adding nothing else.
It's clear that both solutions produce infinite sets. Why is the latter not more simple?
 A: First let me clarify my comment, $\{x\}$ is not in the language of set theory. The language of set theory uses the braces as a crude shorthand. Instead, $\{x\}$ is a shorthand for $\forall y(y\in\{x\}\leftrightarrow y=x)$. This means that to talk about $\{x\}$ one needs to say $\exists z(\forall y(y\in z\leftrightarrow y=x)\land\dots)$, because $\{x\}$, as we said, is not part of our formal language.
When you take bof's suggestion and unpack everything, the result is not significantly shorter. It also has the slight problem that the inductive set we get is the limit, in some sense, of $\varnothing,\{\varnothing\},\{\{\varnothing\}\},\{\{\{\varnothing\}\}\},\dots$ and one might expect it to simply be $\{\dots\{\varnothing\}\dots\}$, where the ellipsis here denote an infinite amount of braces. And that's wrong.
Regardless of that, the Axiom of Infinity is not saying that $\omega$ exists, or that some von Neumann ordinal exist. It states that there is a set with a simply defined property: being an inductive set.
The smallest such set is provably $\omega$, which is why we often thing about the Axiom of Infinity as simply stating "$\omega$ exists". But that is not what it says. The question you link asked why can't we instead postulate something like $\Bbb R$ exists, and the answer is that the property of being inductive is simple, whereas the property of being $\Bbb R$ is not even part of the language of set theory, and while we can formalise a canonical copy of $\Bbb R$, it requires a lot more than stating the existence of inductive sets.
A: I do not think that the first set is considered the simplest solution set that witnesses the axiom of infinity. There are set theory books that use the second set to witness the axiom (e.g. Moschovakis' book "Notes on Set Theory"). It does not really matter, which set you choose though...
The main practical difference between the two sets is that the first one is transitive. It is therefore the smallest infinite ordinal, a fact that has its own practical significance.
A: The simplest infinite set in structure is definitely the set of all Zermelo naturals, i.e. the set of all iterated singletons of the empty set. The set of all Von Neumann naturals is not the simplest in structure, but if one requires a definition of the ordinals that suits many of them being infinite, then its here were Von Neumann's would trip in.
Its funny to see that even for that purpose Von Neumman ordinals are still not the simplest in structure! The simplest would be sets iteratively formed by starting from the empty set then applying the singleton operator at successive stages; and at limit stages taking the set of all sets formed at prior stages. 
What really von Neumann's enjoy is having the SIMPLEST definition of the strict smaller than relation $<$ on ordinals, which is the most important relation about ordinals. This would be set membership itself on ordinals! By comparison, with the other approach, this would be defined as: .. is an element of the transitive closure of.., which is more complex.
Its easier to also well order ordinals horitzontally (i.e., using Von Neumann's) than just doing it vertically (like the above recursively formed ones)
Of course at the finite level, its easier to define the relation successor $\mathcal S$ using the iterative singletons method, since it would be simply $\in$ on ordinals, and $\mathcal S$ is indeed the pivotal relation at that level, but that advantage stops on the shores of the first transfinite ordinal $\omega$. 
Using the infinite kind of iterative singleton formation depicted above, then the relation $\in$ on ordinals would define the following relation on ordinals $$ y=\mathcal S(x) \lor [y \text{ is a limit } \land x < y]$$
The point is that this relation doesn't seem to be pivotal in thinking about ordinals, not at any rate comparable to the relation $<$.
Von Neumann's being transitive is not the basic merit of them, we can have a definition of ordinals that makes them transitive and yet they are not the von Neumann's, for example take ordinals defined recursively starting from the empty set, then we define successors of limit stages (Zero included) by the union of the limit stage and the singleton of it, then we define successor stages of successor stages as: $$\mathcal S(x)= x \cup \{\{max(x)\}\}$$, and take the union of all prior ordinals at limit stages. 
This is less in structure than the von Neumann's, the ordinals defined are transitive, we can define $\leq$ relation on ordinals in a nice manner as the $\subseteq$ relation on ordinals, of course $<$ on ordinals would be defined after proper subset relation on ordinals, however this is more complex than defineing $<$ as simply $\in$ on ordinals.
So the balance goes to Von Neumann's at the infinite realm!
