TL;DR : Set theory is a good thing to know , but there is more math out there which is actually critical for Machine Learning
You should probably learn Math in this order :
1) Pick up this OR this and flirt with the sections on
proofs , set theory and its notations , functions , combinatorics , graphs ; till the time you feel comfortable solving a simple non-trivial problem . Do not pick up books like Halmos , please , -- not discrediting the book in any way - just not required here .
Single Variable Calculus from here and here . Once you understand the intuition behind it , pick up a standard textbook and solve a lot of problems .
3) Learn a bit of
Multi-Variable Calculus from here
Linear Algebra from Strang . Ah ! the joys of learning from Strang . Seriously . You can do yourself no further good . Hop around to other MOOCs like this one .
Probability and Statistics , IMO the most important math field required for ML . There are already pretty good resources and answers on the net , so will not repeat.
6) Learn how to apply statistics , probability theory to real-world problems intuitively by learning
Data Mining and doing competitions on Kaggle
7) Take up Ng's Wonderful Course and embark on a hard yet fun journey of ML
Hope this helps :)
Disclaimer : I am not a ML Researcher , only a CS undergrad who has studied ML and is now working on a ML-based paper under a professor ; so it is possible that the order might vary according to your mileage