For example, if we spend the time to formalize group theory in a set theory (take ZFC as an example), then by the end of this long and arduous exercise, we probably haven't learned a whole lot of new concepts in group theory. We've just worked out how to implement the old concepts in ZFC. Furthermore, we're probably no better at discovering new group-theoretic ideas than we were when we started the whole enterprise. In fact, we might be worse, since now we've (potentially) allowed ourselves to get stuck in the ZFC way of doing things.
The same critique can be directed at any approach to founding mathematics; since its not the only way of doing things, we have to be careful not to let our creativity be stifled by just focusing on that one particular approach. This does not mean we should avoid foundations; rather it means we should always be open to new foundational ideas.
By the way, the main application of set theory to (basic) group theory is that you can actually prove the existence of entities that we take for granted, like Cartesian products of groups. Their existence follows from the existence of Cartesian products of sets, which is a basic result of set theory. (Actually, some approaches take the existence of a Cartesian product of sets as an axiom).
Anyway, the point is this. Just because we can formalize a lot of mathematics using set theory + logic, this does not mean that mathematics $=$ set theory + logic.