The important part is the fact we can at all formalize mathematics in a uniform method.
This means that believing that one theory is consistent, lets you automatically know that everything you can formalize in terms of that theory is consistent as well.
Set theory itself is not particularly important, it is just useful, since it lets us talk about collections of mathematical objects as first-order citizens of the mathematical universe.
Of course, $\sf ZFC$ itself has limitations imposed by the incompleteness theorems, but we can circumvent them by clever tricks such as adding more axioms that will let us talk about set theory from within set theory. Axioms like large cardinals, or the existence of "nice" models of $\sf ZFC$ within our universe.
There are other approaches to formalizing mathematics. Some people think that we have the natural numbers and nothing more, so we can and should formalize things inside some system of arithmetic. Other people prefer type theory, or categories, or other bases for mathematics. But the important thing is that philosophically all mathematics can be done within a particular framework.
So if someone disagrees on the meaning of the word "mathematics", they might have strong opinions against set theory or other types of foundations for mathematics.