I'm learning about some of this right now! As far as I know, you already need quite a lot of set theory to say topos. An (elementary) topos is is a category closed under a lot of common constructions.
These topos gadgets also have an internal logic (usually intuisionistic), and given for instance a natural number object you can perform many of the well-known constructions of mathematics, such as those of the real numbers (here the Cauchy and Dedekind reals may be different). This is what we do in the topos of sets!
An elementary topos $E$, by the way, is a cartesian closed category (which means it has a terminal object $1$, products, exponentiation, and an adjunction between the two, i.e. $\hom_E(X\times Y,Z)\cong\hom_E(X,Z^Y)$) with a subobject classifier $\Omega$ together with a morphism $\top:1\to\Omega$ called true, such that any monic $f:Y\to X$ is the pullback of a unique morphism $\chi(f):Y\to\Omega$. We also require pullbacks along $\top$ to exist. It follows that $E$ has finite limits and colimits, and that the subobject functor $Sub:E\to Set$ is represented by $\Omega$ (i.e. $Sub(-)\cong\hom(-,\Omega)$).