I am a masters student familiar with category theory. I have started learning topos theory from MacLane-Moerdijk's book "Sheaves in Geometry and Logic: A First Introduction to topos Theory". I get the feeling that topos theory has a "logic" part and an "algebraic geometry" part. I am not interested in logic and I am not familiar with algebraic geometry at all. I am though interested in abstract algebra and thus if it has to be one of the two directions, it must be algebraic geometry.
I would like to ask if there is any treatment in the literature of the algebraic geometry needed for a category theorist to become comfortable with notions like the étale and Zariski sites etc. I have heard many people mentioning Grothendieck's EGA when discussing topos theory. But I find this very difficult to read. What do I need to know in order to attempt studying it? Of course in the mentioned book (MacLane-Moerdijk) one can find the elements needed to get an idea of some constructions, but I think that it does not give the big picture. Or do I have to learn algebraic geometry from scratch?