Is the notation for the definition of a set stricly formalized? In the 800 pages set theory book by Jech, he uncommented starts using
$$Y=\{ u\in X : \phi(u)\}$$
as equivalent to
$$Y=\{ u:u\in X \wedge\phi(u)\}$$
on the first few pages. The fact that, in the first version, the expression in the first space before the colon is a formula bothers me a little. 
I wonder if there is are consideration on what the notation $\{ ...\ :\ \ ...\ ...\ ...\}$ is supposed to be doing, such that one of these notation gets forbidden or both are explained. Obviously, it runs through some options, restricted by a condition spacified in the second space. 
How to rigorously formalize $\{ ...\ :\ \ ...\ ...\ ...\}$?
Also, if so, why is it used for sets and also for classes?
 A: First let me reveal a little secret about set theorists, and mathematicians in general: we hate complicated notations. We define different objects in different ways, but we then abuse the notation to ease the readability. It takes time to get used to, yes, but then it gets better.
First note that this is only a notation, if we agree from the beginning that either of these notational conventions mean the same thing, then they indeed mean the same thing. It is possible (and likely, and also true) that sometimes it makes more sense to use one notation and at other times to use the other one.
Second this is indeed a formula, but this is what we call a bounded formula. We bound ourselves to $X$. This often appears in explicit sentences (and formulae) as:


*

*$(\exists x\in X)\varphi(x)$ which really means: $\exists x(x\in X\land\varphi(x))$, or in universal versions:

*$(\forall y\in Y)\psi(y)$ which really means: $\forall y(y\in Y\rightarrow\psi(y))$.


As you can see it saves up both space and improves the readability, especially when there are three quantifiers or more.
Lastly, sets are classes. Recall that a class is a collection defined by a first order formula, but this formula is allowed to have parameters. So $X=\{v\mid v\in X\}$. Not all classes are sets, but all sets are indeed classes. This is why we often use $\{x\in X\colon\varphi(x)\}$, or $\{x\colon x\in X\land\varphi(x)\}$. It tells us that the collection is a subclass of $X$ and if $X$ is a set then this collection is also a set.
To close this, I'll add a remark that one of my most pedantic teachers told me that he sat a long time ago to verify that you can syntactically add the $\{\ldots\}$ to the language and all that. I cannot recall whether he left it undone; tried to retrace his proof a few years after and then left it undone; or originally tried to retrace the proof given by his teacher and left it undone.
A: Yes, I believe it can be done but, in reference to Asaf Karagila's post, I don't know if anyone has ever not left it undone.
ZF set theory is written in the language of first order logic and, the language of first order logic (LFOL) can be extended by definitions. We are interested here in $n$-ary function definitions where $n$ may equal 0 in which case we are dealing with constants.
In LFOL if you have a formula $\phi$ with free variables $x_1,\ldots, x_n, y$ and you have also proven, in a given theory, that
$$
\forall x_1 \ldots \forall x_n \exists!y \phi
$$
Then we may introduce a new $n$-ary function symbol $f$ to the language as well as the definitional axiom
$$
\Gamma_f \equiv \forall x_1\ldots\forall x_n \phi[fx_1\ldots x_n/y]
$$
Following the above procedure, the new theory including the new symbol $f$ and axiom $\Gamma_f$ will be a conservative extension over the original theory.
Let's consider the empty set as an example. The formula which will serve as a defining formula for the empty set is
$$
\phi_{\emptyset} \equiv \forall x (x\not \in y)
$$
The formula $\phi_{\emptyset}$ has one free variable: $y$.
In ZF we can prove the existence of the empty set in a number of ways and the Axiom of Extensionality guarantees the uniqueness of the empty set. In other words it is possible to prove $\exists y (\phi_{\emptyset})$.
We can then introduce the emptyset symbol $\emptyset$ and the axiom
$$
\Gamma_{\emptyset} \equiv \forall x (x\not \in \emptyset)
$$
Set builder notation can be built up using extensions by definition. For example, suppose we want to introduce something like $\{0, 1, 2\}$ into the theory. Such a set can be defined by
$$
\phi_{\{0, 1, 2\}} \equiv \forall x(x\in y \iff (x=0 \lor x=1 \lor x=2))
$$
In ZF if the existence and uniqueness of such a set can be proven (existence can be proven using pairing and uniqueness with extensionality) then we may introduce the new 0-ary function symbol $S_{\{0, 1, 2\}}$ into the theory along with the definitional axiom
$$
\Gamma_{\{0, 1, 2\}} \equiv \forall x (x\in S_{\{0, 1, 2\}} \iff (x=0 \lor x=1 \lor x=2))
$$
Here $S_{\{0, 1, 2\}}$ is the set that we would (in the metalanguage) call $\{0, 1, 2\}$. I use this notation to be a little more consistent with the technical alphabet of LFOL. Of course the alphabet could be extended so that something like $\{0, 1, 2\}$ could be included as a function symbol in that alphabet.
More generally we may want to express a set like $\{u, 2u, v\}$ where $u$ and $v$ are variable In this case the "set" $\{u, 2u, v\}$ is actually a 2-ary function of the variables $u, v$. The defining formula for this set will be
$$
\phi_{\{u, 2u, v\}} \equiv \forall x (x\in y \iff (x=u \lor x=2u \lor x=v))
$$
If we can prove
$$
\forall u \forall v \exists!y \phi_{\{u, 2u, v\}}
$$
Then we may introduce the 2-ary function symbol $S_{\{u, 2u, v\}}$ and the definitional axiom
$$
\Gamma_{\{u, 2u, v\}} \equiv \forall u \forall v \forall x (x\in S_{\{u, 2u, v\}}uv \iff (x =u \lor x=2u \lor x=v))
$$
Note that I've written $S_{\{u, 2u, v\}}uv$ indicating we must "plug" $u$ and $v$ into $S_{\{u, 2u, v\}}$ to get a set which can be a target for set membership $\in$.
All of the above has been for finite sets that result from specifying a finite list of elements. Such sets can be constructed using pairing. This was in preparation for sets of the type pointed out in the OP. The OP inquires about sets of the form
$$
\{x \in A: \psi\}
$$
Such sets are generated by the the specification schema.
the recipe is similar however. Typically such sets will be constants.
The defining formula would be something like:
$$
\phi_{\{x\in A:\psi\}} \equiv \forall x (x\in y \iff (x\in A \land \psi))
$$
The existence and uniqueness of sets $y$ satisfying this formula are guaranteed by specification and extensionality, i.e. we can prove
$$
\exists! y \forall x(x\in y \iff (x\in A \land \psi))
$$
Which means we may introduce the 0-ary function symbol $S_{\{x\in A:\psi\}}$ and the defining axiom
$$
\Gamma_{\{x\in A:\psi\}} \equiv \forall x(x\in S_{\{x\in A:\psi\}} \iff (x\in A \land \psi))
$$
If $\psi$ has free variables $x_1, \ldots, x_n$ then we can extend this to a sort of "parametric" set builder notation as above.
We prove
$$
\forall x_1\ldots \forall x_n \exists!y (x\in y \iff (x\in A \land \psi))
$$
And introduce the $n$-ary function symbol $S_{\{x\in A: \psi\}}$ and definitional axiom
$$
\Gamma_{\{x \in A: \psi\}} \equiv \forall x_1\ldots \forall x_n (x\in S_{\{x \in A: \psi\}} x_1\ldots x_n \iff (x\in A \land \psi))
$$
