Foundations of Mathematics For a long time I've felt that the search for foundations in mathematics doesn't have an answer, that it is almost like physics where you can keep asking where things come from.
However, is this not the case? People say that set theory is the/one foundation for mathematics, but set theory uses logic. Moreover, in logic, and here is my largest concern, we define things like strings of characters and statements with truth value.
These things don't appear to require any other mathematical objects, however it seems somewhat non-rigorous to introduce strings and truth values and characters. Is there a chicken-and-egg issue somewhere here, are these ideas well-formulated, should I be confused? In what sense is set theory the foundation of mathematics?
 A: Ultimately, any proof is a string of characters written on paper. So if we don't believe we can make mathematical arguments about strings of characters written on paper, we can never hope to say anything rigorous about the concept of proof. Therefore, at a minimum, we accept certain arguments about strings of characters, which are equivalent to a kind of arithmetic of "intuitive integers." By an "intuitive integer" I mean any possible length of a string of characters written on paper. This view sees logic as a theory of the syntax of formal proofs, where a formal proof is a string of characters satisfying certain well-defined conditions. Theorems about strings of characters written on paper are proved using what we think we know about intuitive integers; they are not proved within a formalized system. (Of course, within a formal theory we can model the concept of a formal proof actually written in ink, and prove theorems about these proofs. But that is different from talking about something written on a physical piece of paper.)
When you talk about "truth values," you are talking about semantics. The reason we talk about semantics (I'm probably oversimplifying) is that we may wish to convince ourselves that a particular syntactic definition of what constitutes a formal proof is a reasonable one. To talk about semantics requires that we talk about models of a particular theory, hence that we have some concept of "set," "relation," etc., or something that can stand in for these. 
However, in order to discuss formal proofs and to prove theorems about them, it isn't necessary to talk about semantics at all. The semantic aspects of logic merely justify the choice of a particular way of formalizing proofs. For example, the first chapter of Bourbaki's book on set theory defines what a formal proof is (in their system), but never officially mentions the concepts of "true" or "false."
In a very basic sense, if we agree on a particular system of formal proof, then mathematics consists of writing down strings of characters that follow certain rules, and mechanically verifying that those rules have been followed. The rules will have been chosen to be simple enough that even a computer can be programmed to check they've been followed. The difficulties come in only when we ask whether the set of rules makes any sense.
A: DISCLAIMER: Not sure how good or professional this explanation is going to actually be.For all I know I'm totally wrong. All of what I have to say is just my fundamental belief based on what I can conclude from research I have done going into a senior thesis I am working on, as well as general curiosity from trying to answer myself THE VERY QUESTION you have asked. Hopefully this helps.
A formal mathematical statement like "every bounded set has a least upper bound in $\mathbb{R}$" are just words that syntactically behave like a classical predicate logic. And our proofs that we write down on paper and accept in academia follow the "inference rules" that we all know and love. Why? No other reason than "this is how mathematicians talk." Why do mathematicians talk this way? Because it makes sense. We can place rigor in a sensible way that doesn't run us into errors that tear the mathematical system apart.
The point being: These "mathematical objects" you speak of are nothing more than an abstraction. And the whole purpose behind abstractions are to cleverly model the things in a "logical" way so that it "behaves" the way we want it to. For example, set theory came up historically as one of the first modernly recognized "fundamental" systems of classification. When we started seeing (through Cantor) that not all "infinite classifications" (such as $\mathbb{N}$ and $\mathbb{R}$) were equal--despite what great minds like Galileo believed--it became necessity to "keep a tab" on the "elements" belonging to our classifications. And thus set theory was born, and dandy things like "cardinality" with it!
When one says set theory is a "foundation of mathematics", one really doesn't actually mean "THE first principle of mathematics" (although it is certainly easy to think that...only to be initially disappointed once proven otherwise, but I promise you there is a beauty in this "incompleteness" from within). The real grounbraking-ness in set theory was its ability as an abstraction to adequately model things like number systems, algebraic structures, geometric structures, and so on. But set theory is nowhere near close to a "first principle", particularly when you include what logicians would love to call "metamath": Abstractions of abstractions of things that resemble "everyday arithmetic".
For that, you would need something like "the category of all sets" which is "a proper class" as opposed to a set, axiomised using the Von Neumann–Bernays–Gödel (NBG) axioms. And it doesn't stop there. "The category of all classes" is not a class, and we essentially need a replica of the (NBG) axioms for whatever we want to call the "larger object" that contains all classes. And we can go ON forever.
But that's not a problem. Because like I said one doesn't need an actual "first principle" to do math "correctly". All we need is "well-behaved" abstraction to model whatever it is we want to do and then just do it! Whether that abstraction be set theory, first order predicate logic, type theory, universal algebra, etc. Furthermore sometimes it brings us amazing theoretical insight to relate one such abstraction to another (such as real analysis with topology, a synergy which in fact was very intentional by the inventors of topology). But that doesn't mean "one comes exclusively from another" necessarily. It just means "you can use one to explain another," which is not to say there isn't another way. For example, category theory is commonly used to explain "natural deductions" in a language, but my project intends to take a more graph theoretic approach.
Hopefully that makes at least some sense.
