What are the must-know concepts and best resources for preparing the mathematical background for advanced machine learning studies?

Currently, looking into the book What is Mathematics? by Richard Courant to strengthen my fundamentals. Are there any better references that can help? And would it be worth spending time on such basic concepts like number system, congruences etc?

Also, looking for more study material that can help me take a step towards a deeper understanding of the subject towards the discipline of data science and machine learning.

  • $\begingroup$ 2-3 university courses in linear algebra, some probability theory, 2-3 courses in analysis, some numerical analysis, some algorithmics/algebra, etc. A minor in math is a good start. $\endgroup$ – Olivier Mar 1 '18 at 12:31
  • 2
    $\begingroup$ Some highly relevant subjects are linear algebra (I recommend one of Strang's books such as Introduction to Linear Algebra), multivariable calculus (particularly the gradient and the multivariable chain rule), probability (Bertsekas's probability book is not bad), and some optimization (such as gradient descent and stochastic gradient descent). $\endgroup$ – littleO Mar 1 '18 at 12:35

It definitely depends on what you want to do, since ML is a relatively large and diverse field now. A quick summary might be something like this:

Basics (i.e. needed for the more advanced ones below)

  • Linear algebra (e.g. matrix operations and decompositions, vector spaces)
  • Multivariate calculus (e.g. gradients and jacobians for optimization)
  • Basic probability and statistics (e.g. basic distributions & estimators)
  • Algorithmic analysis
  • Basic signal processing (e.g. convolutions, Fourier series)

Mathematical Theory (e.g. PAC theory)

  • Analysis & measure theory (e.g. advanced probability)
  • Functional analysis

Probabilistic Modelling (e.g. Bayesian deep learning, generative modelling)

  • Stochastic processes & information theory (e.g. MCMC, variational inference)
  • Advanced statistics (e.g. properties of estimators, convergence of distributions)

Implementation-Oriented ML

  • Optimization (e.g. convex optimization)
  • Numerical analysis (e.g. discretizations)
  • Computational numerics (e.g. error accumulation, matrix algorithms)

(Just to link some relevant questions on how to study basic ML mathematically to this one: [1], [2], [3], [4], [5], [6], [7], [8], [9], [10] )

| cite | improve this answer | |

Machine Learning as a whole is incredibly diverse. Likewise, the type of math seen largely depends on the certain kind of questions you're interested in. Regardless of your interests, a strong background in linear algebra and probability/statistics is a must.

Assuming that you're interested in Deep Learning (a fair assumption to make considering it's at the height of a huge popularity wave), you will want to make sure you're comfortable with multivariable calculus and some basic optimization (this can be achieved through a strong understanding of calculus and linear algebra).

Deep Learning at a practical level is very accessible to someone with an understanding of early undergraduate mathematics courses. You could really just pick up a book and go through the introductory sections to see what kind of theory the authors utilize. Deep Learning by Goodfellow et al (freely available online) dedicates a third of the book to building up prerequisite math and statistics. It's intended as a review of concepts so you could then supplement those sections with specific texts if you need to.

My biggest recommendation though is to not get too caught up in reading. Since you're interested in data science, you need to actually work with and explore real datasets. When introduced to a new algorithm don't just go through the main ideas, strengths, weaknesses, etc. Implement it for yourself on some reasonable dataset (MNIST is a very popular beginner dataset for computer vision). Deep Learning is still in its infancy as a scientific discipline. The majority of results are coming from intuition. You can only gain this by actually working with these things. Come up with ideas to address issues, run tests, see what works/what doesn't work. You will understand things a lot better this way.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.