EDIT: for a variety of reasons, I think I should give an explicit caveat here. Obviously I believe the things in my answer below - otherwise I wouldn't have written it. But I am sure there are many, many mathematicians smarter than me who would strongly disagree with some point, or even all of it. I think this is an area where it is easy to generate heated discussion, so I want to be clear that everything below reflects my own views, and that I am aware that I am subject to a number of biases, both mathematical and philosophical.
OK, let me clarify what I meant with that remark. Certainly its naive interpretation is blatantly false, and I could have written much better, but I did mean something by it, and I stand by that.
Incidentally, I think it may be possible to read this as anti-set-theory. That's definitely not my position - I consider myself a set theorist (and a computability theorist, but that's less relevant at the moment! I even think that mathematics could benefit from more of the community more seriously engaging set-theoretic issues; what I say below is descriptive, not prescriptive.
It is very well-known that most of mathematics does not care about foundational matters; I think, for example, that the vast majority of mathematicians would be unable to state the axioms of ZFC. And that's fine! Math long predated the emergence of a generally accepted foundation, and there are and will continue to be challenges to that (or any) foundation (I'm speaking in particular of homotopy type theory, about which I know nothing but have heard is very cool and fundamentally not about sets; maybe someone who knows better can step in, here?).
But I claimed something stronger, that most math is not about sets. So in what sense to mathematicians not talk about sets? Well, obviously we do in a sense - e.g. a group is universally defined as "a set such that . . .". My point is that this definition is vague - the notion of what a "set" is here is the naive one, and naive set theory is inconsistent. So while mathematicians use the word set, the use of any precise notion of set is generally not part of mathematical practice - it's accepted that any natural use of sets won't run afoul of the paradoxes of naive set theory, and will be formalizable routinely (if tediously) in, say, ZFC.
So my point is that while mathematicians use the informal concept of "set" all over the place, most of the time we do not use any precise notion of "set". But it gets worse: there are widely-used parts of math which are fundamentally non-set-theoretic, in the sense that casting them in set-theoretic terms is extremely unnatural: e.g. do you really think of a real number as an equivalence class of sequences of rationals? Demanding a set-theoretic background means abandoning structuralism; as a set theorist, I approve of this, but I also recognize that in doing so I'm breaking with a large number of mathematicians, if not the majority. See also here and here.
What I claim, in short, is the following: although the informal concept of "set" is of course universal in math, to very large extent math is opposed to working in any specific formalization of the concept, instead accepting as a matter of practice that the portion of naive set theory that is invoked will be consistent. In particular, lots of mathematical practice is fundamentally non-set-theoretic.