Most set theories, such as ZFC, require an underlying knowledge of first-order logic formulas (as strings of symbols). This means that they require acceptance of facts of string manipulations (which is essentially equivalent to accepting arithmetic on natural numbers!) First-order logic does not require set theory, but if you want to prove something about first-order logic, you need some stronger framework, often called a meta theory/system. Set theory is one such stronger framework, but it is not the only possible one. One could also use a higher-order logic, or some form of type theory, both of which need not have anything to do with sets.
The circularity comes only if you say that you can justify the use of first-order logic or set theory or whatever other formal system by proving certain properties about them, because in most cases you would be using a stronger meta system to prove such meta theorems, which begs the question. However, if you use a weaker meta system to prove some meta theorems about stronger systems, then you might consider that justification more reasonable, and this is indeed done in the field called Reverse Mathematics.
Consistency of a formal system has always been the worry. If a formal system is inconsistent, then anything can be proven in it and so it becomes useless. One might hope that we can use a weaker system to prove that a stronger system is consistent, so that if we are convinced of the consistency of the weaker system, we can be convinced of the consistency of the stronger one. However, as Godel's incompleteness theorems show, this is impossible if we have arithmetic on the naturals.
So the issue dives straight into philosophy, because any proof in any formal system will already be a finite sequence of symbols from a finite alphabet of size at least two, so simply talking about a proof requires understanding finite sequences, which (almost) requires natural numbers to model. This means that any meta system powerful enough to talk about proofs and 'useful' enough for us to prove meta theorems in it (If you are a Platonist, you could have a formal system that simply has all truths as axioms. It is completely useless.) will be able to do something equivalent to arithmetic on the naturals and hence suffer from incompleteness.
There are two main parts to the 'circularity' in Mathematics (which is in fact a sociohistorical construct). The first is the understanding of logic, including the conditional and equality. If you do not understand what "if" means, no one can explain it to you because any purported explanation will be circular. Likewise for "same". (There are many types of equality that philosophy talks about.) The second is the understanding of the arithmetic on the natural numbers including induction. This boils down to the understanding of "repeat". If you do not know the meaning of "repeat" or "again" or other forms, no explanation can pin it down.
Now there arises the interesting question of how we could learn these basic undefinable concepts in the first place. We do so because we have an innate ability to recognize similarity in function. When people use words in some ways consistently, we can (unconsciously) learn the functions of those words by seeing how they are used and abstracting out the similarities in the contexts, word order, grammatical structure and so on. So we learn the meaning of "same" and things like that automatically.
I want to add a bit about the term "mathematics" itself. What we today call "mathematics" is a product of not just our observations of the world we live in, but also historical and social factors. If the world were different, we will not develop the same mathematics. But in the world we do live in, we cannot avoid the fact that there is no non-circular way to explain some fundamental aspects of the mathematics that we have developed, including equality and repetition and conditionals as I mentioned above, even though these are based on the real world. We can only explain them to another person via a shared experiential understanding of the real world.