It depends on what type of QM course you want to take. Courses in QM for engineers, undergraduate physics majors, graduate students in physics, and graduate students in mathematics are all pretty different. I will assume you're seeking an "undergraduate physics major" understanding of QM.
I have two recommendations:
In my opinion, the best Quantum Mechanics book for self-study is Shankar. My main reason is that the first third or so of the book is a survey of the mathematics you'll need in the other two thirds, i.e., it answers precisely the question you've asked. You'll need to know calculus first (including vector calculus), but from there Shankar will give you what you need. I also like this book because it includes tons of fully worked out examples, and pages upon pages of "what does this mean" type exposition. Usually, you'll see this book being used in graduate or advanced undergraduate quantum mech courses, but that is just because it is a very long book about a very involved subject, it doesn't mean it isn't accessible to the beginner.
When I took quantum mech I read Lang's Linear Algebra concurrently, and it made everything so, so much easier. You don't necessarily have to finish it, basically just get real comfortable with inner products and dual spaces, up through the spectral theorem.
Some additional remarks:
I think it's an exaggeration to say a course in quantum mechanics requires functional analysis or operator theory, even though that is essentially what you're doing. The mathematically rigorous forms of those disciplines are pretty advanced, but you don't need to understand them at that level to do quantum mechanics. You just need to be able to use them. Shankar will teach you how to do that. (Of course, I wouldn't discourage you from learning them eventually, but they aren't strictly necessary for the purposes of QM.) He should catch you up on the basics of probability theory, too.
I would not say that about abstract linear algebra, however. You will need to understand vector spaces, dual bases, inner products, eigenvalues, etc. on a rigorous level.
Group theory and representation theory are definitely not necessary. Although they are important to quantum mechanics, you will not see them until very advanced levels.
So, long story short: you need to know calculus up through vector calculus and linear algebra up through abstract linear algebra.
It's worth mentioning that, although what I've mentioned above would be sufficient for the mathematics side of things, it would be real good if you had seen mechanics, electricity and magnetism, and thermodynamics beyond an introductory level beforehand. It's possible to do it without previous physics courses, but you'll find yourself saying "so what?" a lot, as the weirdness of quantum mech will not seem as jarring to you if you have not seen how things work at larger scales.