9
$\begingroup$

I am a math master student and have done fundamental math courses like probability theory, measure theory, linear algebra and know a little bit about functional analysis. What is good way for me to learn machine learning in depth?

I have read the classical text Pattern-Recognition and Machine Learning last summer; my impression was that it was very ineffective to read the book chapter by chapter like a mathematical text. The book does not go deep enough for many algorithms and skip too many steps considered too technical by engineers.

Is there a machine learning book that maybe does not cover too many topics, but treat each one in depth and takes advantage of math when necessary? It will be great to be able connect fundamental mathematical objects with machine learning (I am thinking about Lp spaces, hilbert space etc).

$\endgroup$
3
  • $\begingroup$ not sure you need a book. did you look at kernel k-means, kernel-PCA, kernel-Fischer discriminant, SVM, spectral clustering ? $\endgroup$ – reuns Jul 13 '16 at 0:09
  • $\begingroup$ I know some of them pretty well like svm and kernel PCA. Why do you ask? $\endgroup$ – user1559897 Jul 13 '16 at 0:12
  • $\begingroup$ well then you know the theory. all you have to do now is reading papers and downloading programs for seing how things work really (pagerank, OCR, object tracking, neural networks, etc.) $\endgroup$ – reuns Jul 13 '16 at 0:13
5
$\begingroup$

Hmm, that is a very good question!

I see a couple avenues you could proceed:

  1. In general, once you start doing rigorous machine learning, the distinction with modern statistics really starts to vanish. So you could pick up a book from that literature. I would recommend for example, the book by Peter Bühlmann and Sara van de Geer. It essentially only considers one model: The LASSO (as well as slight variations, such as L1-penalized logistic regression), and is a "math" book (lots of definitions, lemmata, theorems, proofs).

  2. Keeping with the idea of only studying one method, you could read the book on boosting by Schapire and Freund. This would give a flavour of rigorous results, but with more of a CS rather than stats perspective. [Caveat: I have not actually read this one, but have heard good things about it.]

  3. The two previous recommendation focus on understanding one specific method (L1 regularization, boosting respectively); if instead you want a unifying framework in terms of a specific mathematical space, then there is a very nice review paper of RKHS (reproducing kernel Hilbert space) machine learning methods in the Annals of Statistics. It's not a textbook, but it seems to perfectly suit your needs.

  4. Larry Wasserman has been teaching an amazing course on Statistical Machine Learning. There he goes through a whole bunch of unrelated methods (what you complained about), but for each one he explains what the main mathematical tools and results are, and proves many of these. The website provides both videos of the lectures, exercises and lecture notes (which I think he is compiling into a textbook). Highly recommended.

$\endgroup$
3
$\begingroup$

I think, your difficulty arises from being used to developed unified theories (e.g., theory of bounded linear operators on Hilbert spaces), whereas Machine Learning is no such thing. It is, rather, a collection of (classes of) techniques, most based on optimization of some sort.

So, I would start with reading the individual Wikipedia articles on the different techniques and areas of Machine Learning: regression, logistic regression, Principal Component Analysis, Support Vector Machines, Vapnik-Chervonenkis theory, deep learning, and nonlinear dimensionality reduction.

If you want to connect these to fundamental mathematical objects, then there are these articles, but the objects have more to do with differential geometry than with functional analysis:

*) Smale et. al., ''Finding the homology of submanifolds with high confidence'' *) R. Ghrist, “Three examples of applied and computational homology," Nieuw Archief voor Wiskunde 5/9(2).

$\endgroup$
3
  • $\begingroup$ I totally agree with you that machine learning is a collection of (classes of) techniques, most based on optimization of some sort. And, I do know that there are deeper connections among many learning algorithms. For example, SVM has a strong connection with statistical learning theory and the kernel trick has wide applications in a number of algorithms. With these in mind, I could not help myself to find out the connections, but has not discovered the answers so far in the books I have. Not sure what I should do to learn things like above mentioned. $\endgroup$ – user1559897 Jul 13 '16 at 0:22
  • 2
    $\begingroup$ @user1559897 : I do not agree. the theory I talked above is that machine learning (as statistics) is based on very simple models, simple optimization criterion, and some algorithms for solving it, with the help of a few mathematical tricks. but the most important is that when your data follow a very simple model, it works, and when the true underlying model is more complicated, it doesn't. that's the only one deep theoretical fact you need to know in machine learning :) $\endgroup$ – reuns Jul 13 '16 at 0:51
  • $\begingroup$ In that case, maybe start with this website: cs.huji.ac.il/~tishby $\endgroup$ – avs Jul 13 '16 at 2:07
2
$\begingroup$

With a background in pure math you will surely enjoy these books:

Mohri's Foundations of Machine Learning which is available for free

Shalev-Shwartz's Understanding Machine Learning: From Theory to Algorithms also available for free

Devroye's A Probabilistic Theory of Pattern Recognition

Lattimore's Bandit Algorithms

You will also like the publications of Foundations and Trends in Machine Learning.

$\endgroup$
-2
$\begingroup$

There is a good book that is probably easier to start with: An introduction to statistical learning, by hastie and tibshirani.

I also have an online course that will go through the basic topics in machine learning. You can find out more by checking out:

Math for Machine Learning Online Course

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.