It depends which areas of the philosophy of mathematics you want to study. We can usefully divide the field into (A) very general Big Picture questions about how mathematics fits into our general views about the world and our knowledge of it; and then there are (B) more specific questions that arise from reflecting on some of the details of mathematical practice. You don't need a lot of mathematical knowledge to tackle the first sort of question (though you need a lot of other philosophical knowledge); you need more, perhaps a great deal more, mathematical knowledge (but less general philosophy) to tackle the second sort of question.
Let me spell that out a bit (with the preliminary remark that I wouldn't want to say that there is a really sharp division here: still, I suggest that it is a very useful first approximation to think in terms of there being two different sorts of question here.)
(A) There’s a lovely quote from the great philosopher Wilfrid Sellars that many modern philosophers in the Anglo-American tradition [apologies to those Down Under and in Scandinavia ...] would also take as their motto:
The aim of philosophy, abstractly formulated, is to understand how
things in the broadest possible sense of the term hang together in the
broadest possible sense of the term.
Concerning mathematics, then, we might wonder: how do the abstract entities that maths seems to talk about fit into our predominantly naturalistic world view (in which empirical science, in the end, gets to call the shots about what is real and what is not)? How do we get to know about these supposed abstract entities (gathering knowledge seems normally to involve some sort of causal interactions with the things we are trying to find out about, but we can’t get a causal grip on the abstract entities of mathematics)? Hmmmm: what maths is about and how we get to know about it — or if you prefer than in Greek, the ontology and epistemology of maths — seems very puzzlingly disconnected from the world, and from our cognitive capacities in getting a grip on the world, as revealed by our best going science. And yet, … And yet maths is intrinsically bound up with, seems to be positively indispensable to, our best going science. That’s odd! How is it that enquiry into the abstract realms of mathematics gets to be empirically so damned useful? A puzzle that prompted the physicist Eugene Wigner to write a famous paper called “The Unreasonable Effectiveness of Mathematics in the Natural Sciences”.
Well, perhaps it’s the very idea of mathematics describing an abstract realm sharply marked off from the rest of the universe — roughly, Platonism (for a short-hand label) — that gets us into trouble. But in that case, what else is mathematics about? Structures in some sense (where structures can be exemplified in the non-mathematical world too, which is how maths gets applied)? — so, ahah!, maybe we should go for some kind of Structuralism about maths? But then, on second thoughts, what are structures if not very abstract entities, after all? Hmmmm. Maybe mathematics is really best thought of as not being about anything “out there” at all, and we should go for some kind of sophisticated version of Formalism -- perhaps it is all just symbol shuffling, that doesn't have to hook up to some abstract Platonic reality).
And so we get swept away into esoteric philosophical fights, as the big Isms slug it out. Well, I caricature of course! -- but the key point is that you don't need to know a great deal of advanced maths to follow these fights, they arise from quite elementary reflections on the school-room beginnings of maths and on the applications of elementary mathematics. But to get anywhere, you do need to be able to follow arguments in general metaphysics and epistemology, i.e. follow arguments about what there is and about how we can know about it.
(B) However, philosophers of mathematics talk about much more than this Big Picture stuff. To be sure, the beginning undergraduate curriculum in the philosophy of mathematics tends to concentrate in that region: e.g. for an excellent textbook see Stewart Shapiro’s very readable Thinking about Mathematics (OUP, 2000). But the philosophers also worry about more specific questions like this: Have we any reason to suppose that the Continuum Hypothesis has a determinate truth-value? How do we decide on new axioms for set theory as we beef up ZFC trying to decide the likes of the Continuum Hypothesis? Anyway, what’s so great about ZFC as against other set theories (does it have a privileged motivation)? In what sense if any does set theory serve as a foundation for mathematics? Is there some sense in which topos theory, say, is a rival foundation? What kind of explanations/insights do very abstract theories like category theory give us? What makes for an explanatory proof in mathematics anyway? Is the phenomenon of mathematical depth just in the eye of the beholder, or is there something objective there? What are we to make of the reverse mathematics project (which shows that applicable mathematics can be founded in a very weak system of so-called predicative second-order arithmetic)? Must every genuine proof be formalisable, and if so, using what grade of logical apparatus? Are there irreducibly diagrammatic proofs? …
That's only the beginnings of a list which could go on. And on. But the point is already made. These questions, standing-back-a-bit and reflecting on our mathematical practice, can still reasonably enough be called philosophical questions (even if they don’t quite fit Sellars’s motto). They are more local than the Big Picture questions which arise from looking over our shoulders and comparing mathematics with some other form of enquiry and wondering how they fit together. These are questions are internal to the mathematical enterprise, discussed by mathematically informed philosophers, as well as by philosophically minded mathematicians -- and you do need varying amounts of serious mathematics to tackle them. You can't, for an obvious example, discuss the foundational significance of category theory if you know no category theory.
So, in summary, you don't need a lot of mathematics to follow debates on some central Big Picture ontological and epistemological questions in the philosophy of mathematics. But other areas of the philosophy of mathematics focus in on specific areas of mathematical practice, and then you do have to know quite a bit of maths to follow them.