What is the advantage of using first-order logic over second-order logic? Second-order logic is more expressive and there is also a way to overcome Russell's paradox...
So what makes first-order logic standard for set theory?
First order logic has the completeness theorem (and the compactness theorem), while second-order logic does not have such theorems.
This makes first-order logic pretty nice to work with. Set theory is used to transform other sort of mathematical theories into first-order.
Let us take as an example the natural numbers with the Peano Axioms. The second-order theory (replace the induction schema by a second-order axiom) proves that there is only one model, while the first-order theory has models of every cardinality and so on. Given a universe of set theory (e.g. ZFC), we can define a set which is a model of the second-order theory but everything we want to say about it is actually a first-order sentence in set theory, because quantifying over sets is a first-order quantification in set theory.
This makes set theory a sort of interpreter, it takes a second-order theory and says "Okay, I will be a first-order theory and I can prove this second-order theory." and if we have that the set theory is consistent then by completeness it has a model and all the higher-order theories it can prove are also consistent.
To read more: