I will try to answer just one of the questions:
in what way does [set theory] try to set up the foundations of mathematics?
It should help to see specific examples of how concepts from other areas of math can be formalized in set theory. Let's consider two examples.
(1) A group can be considered as an ordered pair of objects $(G, f)$ where $G$ is a set and $f$ is a function $G \times G \to G$ satisfying certain logical properties. The Cartesian product $G \times G$ is the set of ordered pairs of elements of $G$. A function $G \times G \to G$ is a subset of the Cartesian product $(G \times G) \times G$ satisfying certain properties. The notion of "ordered pair" can be formalized in terms of (unordered) sets using Kuratowski's definition.
(2) A real number can be defined as a set of rational numbers satisfying a certain logical property (that of being a Dedekind cut.) A rational number in turn can be defined as a set of ordered pairs of integers, namely an equivalence class under a certain equivalence relation. Similarly, integers can be defined as equivalence classes of ordered pairs of natural numbers. Natural numbers can be defined as certain kinds of von Neumann ordinals, which are sets.
In both cases (1) and (2) the reductions to sets are not very elegant, but they get the job done. I think it is remarkable that we are able to do such a thing at all. You could try to reduce set theory and group theory to analysis instead, or perhaps reduce analysis and set theory to group theory, but I don't think you would find as much success.
Of course I don't claim to have shown here that some other approach to the foundations of mathematics, such as category theory, would not work just as well.