Is most of mathematics independent of set theory? Is most of mathematics independent of set theory?  Reading this quote by Noah Schweber:

most of the time in the mathematical literature, we're not even dealing with sets!

it seems that the answer to my question is "yes". But why? When I read in the mathematical literature, sets appear everywhere – we need them in the definitions of groups, rings, vector spaces, ... However, is there some truth in the quote, and in what sense?
 A: EDIT: for a variety of reasons, I think I should give an explicit caveat here. Obviously I believe the things in my answer below - otherwise I wouldn't have written it. But I am sure there are many, many mathematicians smarter than me who would strongly disagree with some point, or even all of it. I think this is an area where it is easy to generate heated discussion, so I want to be clear that everything below reflects my own views, and that I am aware that I am subject to a number of biases, both mathematical and philosophical.
OK, let me clarify what I meant with that remark. Certainly its naive interpretation is blatantly false, and I could have written much better, but I did mean something by it, and I stand by that.
Incidentally, I think it may be possible to read this as anti-set-theory. That's definitely not my position - I consider myself a set theorist (and a computability theorist, but that's less relevant at the moment! I even think that mathematics could benefit from more of the community more seriously engaging set-theoretic issues; what I say below is descriptive, not prescriptive.

It is very well-known that most of mathematics does not care about foundational matters; I think, for example, that the vast majority of mathematicians would be unable to state the axioms of ZFC. And that's fine! Math long predated the emergence of a generally accepted foundation, and there are and will continue to be challenges to that (or any) foundation (I'm speaking in particular of homotopy type theory, about which I know nothing but have heard is very cool and fundamentally not about sets; maybe someone who knows better can step in, here?).
But I claimed something stronger, that most math is not about sets. So in what sense to mathematicians not talk about sets? Well, obviously we do in a sense - e.g. a group is universally defined as "a set such that . . .". My point is that this definition is vague - the notion of what a "set" is here is the naive one, and naive set theory is inconsistent. So while mathematicians use the word set, the use of any precise notion of set is generally not part of mathematical practice - it's accepted that any natural use of sets won't run afoul of the paradoxes of naive set theory, and will be formalizable routinely (if tediously) in, say, ZFC.
So my point is that while mathematicians use the informal concept of "set" all over the place, most of the time we do not use any precise notion of "set". But it gets worse: there are widely-used parts of math which are fundamentally non-set-theoretic, in the sense that casting them in set-theoretic terms is extremely unnatural: e.g. do you really think of a real number as an equivalence class of sequences of rationals? Demanding a set-theoretic background means abandoning structuralism; as a set theorist, I approve of this, but I also recognize that in doing so I'm breaking with a large number of mathematicians, if not the majority. See also here and here.
What I claim, in short, is the following: although the informal concept of "set" is of course universal in math, to very large extent math is opposed to working in any specific formalization of the concept, instead accepting as a matter of practice that the portion of naive set theory that is invoked will be consistent. In particular, lots of mathematical practice is fundamentally non-set-theoretic. 
A: You can think of set theory as a low-level programming language, like Assembly; it works directly with sets in a way analogous to how Assembly works directly with bits and bytes. Some people work with low-level programming languages, like the people who need to write compilers or operating systems or whatever (I am totally making this up, not being a programmer myself), but most people are content to work with high-level programming languages, like Python. These are in some sense ultimately built on low-level programming languages (something somewhere has to actually execute the Python interpreter), but 


*

*the low level has been abstracted away so you don't have to think about it, the same way you don't need to learn how your operating system works in order to program in Python, and

*the low level could be replaced by something else, and as long as it's still good enough to support the high level, it doesn't really matter, the same way you can program in Python on both a Windows machine and a Mac. 
So yes, in some sense it's true that when you're doing mathematics you're (usually) secretly dealing with sets, in the same way that in some sense it's true that when you're programming you're secretly manipulating bits in your physical computer. But most people neither want nor need to think all the way down the layers of abstraction like that, because / and it doesn't matter what the bottom layer of abstraction looks like as long as it can run the layers above it. 
A: Most mathematics can be translated into some suitable set theory such as ZFC. That is certainly true! However, it is totally different from the claim that most mathematics deals with sets. From the point of view within ZFC, of course it is true because in ZFC there are nothing else except sets! But in fact, most ordinary mathematics can be translated into a very very weak system called ACA, which has no internal notion of sets of sets. Furthermore, there are alternative systems such as type theories that some logicians even argue to be more natural than ZFC as a foundation for mathematics. That is naturally a subjective opinion, but the objective fact is that there are indeed different formal systems that can 'do the same thing', so to speak, even though one might 'think' of everything as sets while another 'thinks' that there are urelements.
Ultimately I agree with the main point in Qiaochu Yuan's answer. Namely that mathematics is nearly always based on existing structures that obey some properties. For elementary number theory, we do not care what are the internals of each natural number as long as the collection of natural numbers together with the arithmetic operations on them, as a whole, satisfy the Peano axioms. This abstraction would be a valid reason to say that most of them when we use natural numbers we do not actually deal with sets, since it does not matter even if natural numbers are urelements!
Of course, this mirrors the idea in programming that we write high-level programs and do not deal with CPU instructions in the sense that we do not actually care how our programs are translated to CPU instructions. We only care that our program behaves in a way that is solely determined by the high-level programming language. Take Java for example. On different machines it is necessarily translated into different CPU instructions, but that is done by the Java environment; in our terms the Java language is the abstraction that frees us from 'bare-metal' concerns. In the same way our mathematical axiomatizations (like PA for natural numbers) frees us from foundational concerns.
Finally, there is a huge benefit to working with abstractions than with underlying implementations. In case we ever wish to use a different foundational system for whatever reason, abstraction makes it far easier to identify which parts transfer over without change to the new system. We can in fact precisely reason about such general transfer via interpretability.
