That's set theory. I honestly recommend that you study set theory, as nowdays all mathematics are built on sets. Just pick any introductory book to set theory, that ill be (imo) enough.
Now, what you may having difficulties with, is the interpretation of those things, let's take an example from that page:
Given the set of N observations Y = {yi ; i = 1, . . . , N } we want to identify which
observations belong to the same object.
This basically mean that $Y$ is a set containing observations, more simple (not so formal) translation could be: "We have a bunch of observations".
We are looking for a partition ω of observations from Y into several
trajectories Yk ⊂ Y (subsets of Y ) such that each trajectory collects
observations believed to come from a single person.
This now means that $\omega$ (defined in the next parragraph) is a partition from a space of partitions, and you're looking for subsets, that they call $Y_k$ that is believed to belong to the same person. Now the translation: "We are looking for observations that we believe belong to the same person, and we are going to separate the observations that belong to every person in different groups (sets)".
A valid partition expresses the set of all observations as an
exhaustive union of trajectories: Y = Y1 ∪ . . . ∪ YKω , where Kω is
the number of objects proposed by a partition ω. The trajectories must
be mutually exclusive: Y ∩ Yk = ∅, when = k.
And last, they say a valid $\omega$ must have trajectories that contain (alltogether) the initial set $Y$, and that this trayectories' intersection must be empty. The translation: "The partition must have all subsets of observations, that means we must have formed groups that cover all observations, and two different subsets of observations cannot share a member, that would be obviously absurd, if we had two belonging to person A, and another two belonging to person B, and one of those two was repeated, then A and B would be the same person"
That goes on for the whole article. The final parragraph means:
We have a bunch of observations. We are looking for observations that
we believe belong to the same person, and we are going to separate the
observations that belong to every person in different groups (sets),
that separation is called "partition". The partition must have all
subsets of observations, that means we must have formed groups that
cover all observations, and two different subsets of observations
cannot share a member, that would be obviously absurd, if we had two
belonging to person A, and another two belonging to person B, and one
of those two was repeated, then A and B would be the same person.
This notation is used because it doesn't let space for ambiguity, and is more rigouruos, so besides studying set theory, which is basic, you should train you interpretation of these articles (just read more and try to do what I've done here)
BTW, I don't think there's any book about mathematical notation for computer science (it would really surprise me), because such thing doesn't exist, that's just mathematical/formal notation.
