I strongly empathize with your questions. However, you are unlikely to get satisfactory answers on a site like this. One reason I say this lies with the distinction you draw between "real" mathematics and studies in mathematical logic. It may be the case that what someone holds to be mathematics depends greatly upon what they have been taught under the guise of mathematics. A great deal of mathematics is taught in philosophy departments. So people who had received their initial training in advanced mathematics in such an environment will not even acknowledge the distinction you seem to be making.
I have studied the foundations of mathematics as an avocation for the last 35 years. I received my undergraduate training in the mathematics department of the University of Chicago. Their mathematics department is a part of their Physical Sciences Collegiate Division. So, you may surmise that my understanding of "real" mathematics is not derived from philosophical methods. I had been accepted to graduate programs at schools like Princeton, Harvard, Yale, and UC Berkeley. Unfortunately, a personal tragedy ended any hope of an academic career. I never received the benefit of a graduate program.
I had been intrigued with the continuum hypothesis. I had begun with the assumption that the applicability of mathematics to the sciences indicated that "real" mathematics was largely unproblematic. So, something was probably wrong with what I had been taught in my mathematical logic courses. After all of this time, I have learned that I think that mathematics ought to be constructive. My use of that word is very different from how those who understand the various foundational paradigms would interpret it. But, my point is that there is an aesthetic component to what someone will judge to be foundational with respect to mathematics.
You ask what a foundation might be if it is not a mere formalism. Setting aside the claims of entrenched paradigms, foundational analysis ought to be directed at discerning and exposing presuppositions assumed in the practice of mathematics. I personally discount the reduction of mathematics to linguistic forms. But, there can be no question that the practice of paraphrasing statements and using formal methods to represent mathematical assertions has enriched the field. The entrenched paradigms carry specific claims with respect to the nature of mathematics.
What you have said about the singleton with only the empty set as a member is simply untrue for a logicist committed to the views of Frege and Russell. Logicism is a metaphysical foundation. To say "mathematics is extensional" is ultimately at odds with the assertion that mathematics is formal. Frege had been critical of formalism, and, Russell challenged the formalists to give a meaningful account of numerical succession. Frege had actually done that using inclusive disjunction. A fundamental difference between Frege and Russell, however, lay with the fact that Frege's account of truth had not been classical (an aesthetic difference with historical consequences). Yet, today Frege's account of zero is seen to be an application of the principle of indiscernibility of non-existents from negative free logic. And once this is understood and recognized, one sees that this principle is ubiquitous in mathematics. Where it is applied, it establishes well-formedness conditions.
To the best that I can ascertain, this is ignored in the study of formal systems because formal systems stipulate a number of well-formedness conditions externally with respect to what modern formalists call an object language. I first became aware of this when reading Universal Algebra by Graetzer. The development of formal systems arises from the pursuit of a logical calculus. Clearly, mathematicians communicate with proofs. So, one might easily pursue foundational studies from the perspective that logic is presupposed by mathematicians. Of course, when I work on a proof, I engage in "synthesis" when asking what follows from the asserted assumptions and "analysis" when assuming the conclusion in order to consider possible antecedents. But, the syntactic structure of the completed proof only presents the synthetic activity. Some might hold that this is all there is to mathematics.
Rudolph Carnap is credited with the distinction between syntax, semantics, and pragmatics. Only pragmatics takes the relationship of the "language user" into account. The formal methods you have questioned concern themselves only with syntax and semantics. I would attribute this to the idea that mathematics has some profound relationship to material reality. This idea certainly influenced a great deal of foundational thought in the late nineteenth century and early twentieth century. While the beliefs of nineteenth century scholars might seems irrelevant, the apparent failure of unwarranted beliefs about geometry led to the arithmetization of mathematics.
Personally, I need a modern argument to convince me that geometry is useless in the study of foundations. You will often find this kind of statement made by advocates of certain entrenched paradigms. In so far as unary negation is eliminable, the 16 basic Boolean functions associated with propositional semantics relate to one another as points in a finite affine plane. Negation is then understood as a particular collineation. Along similar lines, the order-theoretic presentation of logical constants is nearly isomorphic with the order-theoretic presentation of a tetrahedron in combinatorial topology. Actually, though, the comparison should be made with a 3-dimensional projection of a tesseract. I see geometry because no one taught me to disparage geometry.
You mention metalanguages at the end of your question. If, in fact, mathematics begins with axioms, what significance does any metamathematics really have for "real" mathematics? De Morgan's work on symbolic algebras is clear with regard to meaningless symbols manipulated according to stipulated rules of operation. The only exception he makes rests with the sign of equality. Given an informative equality statement (x=y), a substitution may be performed. What justifies a substitution -- that is, its warranting -- necessarily presupposes "meaning." The law of identity provides such a meaning for reflexive equality statements. But, the necessary truth of reflexive equality statements does not follow from a semantic conception of truth. In this sense, a semantic conception of truth reaches back to Frege's account of truth. In so far as formal systems relate to the paraphrasing of natural language, the necessary truth of reflexive equality statements enforces the uniformity of interpretation across occurrences. Of course, this conveniently coincides with the metaphysical law of identity. However, the linguistic mandate follows from what is called the analytic conception of truth. One way of understanding formal systems is to understand the syntactic component as complying with an analytic conception of truth before a semantic conception of truth is applied to its terms. It is a well-formedness presupposition of an entrenched paradigm that is simply stipulated to be mathematics. But, whether or not it is "real" mathematics is a matter of aesthetics.
Supposing that your "real" mathematics attributes some notion of truth to mathematical statements, the idea that meaning is conveyed by truth conditions alone will force you to consider metalanguages. And, questions about truth in the initial metalanguage will force you to accept an unending hierarchy of metalanguages. Unless you are willing to forego your attribution of truth to mathematical statements, you will be forced to concede that "common-sense" has led you to philosophical considerations you would prefer to avoid. If you think that self-consistency is the answer, you should look at what is called the coherence theory of truth. Realists have many legitimate objections to such accounts of truth.
My own aesthetic differs by virtue of a distinction that I have found only in the work of Aristotle. In his syntactic analysis (Prior Analytics) he distinguishes "demonstrations" as a subclass of "deductions." In his further analyses of logic (Posterior Analytics, Topics, and Rhetoric) he distinguishes between "demonstrative argumentation" and "dialectical argumentation." In so far as the notion of a "first principle" applies, it relates to demonstrative logic. The experience of learning mathematics through its careful use of definition corresponds with the ordered presentation of "knowledge statements" that characterizes Aristotle's account of demonstrative argumentation (Categories). But, the emphasis on semantics in the modern development of mathematical logic is aligned with Aristotle's dialectical argumentation. Since my axiom systems require different quantifier rules to capture both, I see foundations as a branched system.
In addition, my aesthetic motivation leads to the idea that "foundational theories" are necessarily intensional. As noted above, one must forego one's attribution of truth to mathematical statements to avoid hierarchies of metalanguages.
Hopefully, you will derive some benefit from these remarks. It takes a great deal of effort to read and (hopefully) understand ideas with which one does not agree. Because of what economists call opportunity cost, my studies in foundations have taken away from ordinary mathematics which I would have liked to have learned. But, the questions which have led to different foundational programs are well-motivated. I have not suffered for my choices beyond the ad hominem attacks I have experienced on sites like this.