2
$\begingroup$

I only have a German high school education in mathematics. What mathematical courses/books/concepts would you recommend me to study before reading "deep learning" by Ian Goodfellow? My high school education introduced me to things like vectors, matrices or calculus only on a purely task-based level, but I don't have a formal education in these subjects, meaning I am mostly unfamiliar with their abstract mathematical notations and deeper conceptual backgrounds you would only learn at university.

I had a look at this thread: Which course for deep learning? but the questioner already had a better mathematical background as me, it seems.

What do I need to understand in the Goodfellow book? I would like to understand deep neural networks on a conceptual as well as a mathematical level. I am not interested in programming.

$\endgroup$
6
  • 1
    $\begingroup$ You can't understand deep learning without having a basic understanding of some university level stuff like linear algebra and probability theory. Fortunately, many of these courses are taught online for free on many platforms like coursera, eduX, etc. You will need a minimum of 2 years of preparation before you can understand the mathematics of deep learning. $\endgroup$ Jan 27, 2019 at 14:03
  • $\begingroup$ Have a Master degree in Computer Science, first. $\endgroup$
    – Wuestenfux
    Jan 27, 2019 at 14:05
  • 1
    $\begingroup$ @Wuestenfux I'm a 3rd year undergrad student in pure mathematics and understand most of the mathematics involved in machine learning and deep learning without much effort. I don't think an M.Sc. degree is necessary, specially in Computer Science. $\endgroup$ Jan 27, 2019 at 14:12
  • $\begingroup$ Deep learning alone? No. You need to get the big picture such as support vector machines, bayesian networks, fuzzy logic and whatnot. $\endgroup$
    – Wuestenfux
    Jan 27, 2019 at 18:03
  • 1
    $\begingroup$ @Wuestenfux You can understand a good deal of support vector machines, Bayesian networks, probabilistic graphical models, CNNs, RNNs, GANs, the Boltzmann machine, DBNs or other topics just by knowing linear algebra, probability theory and calculus well. That's what engineers do and most of the progress in implementing these things in real life has been done by them. It's not like quantum field theory (QFT) that requires a deep knowledge of different branches of mathematics. $\endgroup$ Jan 27, 2019 at 18:24

1 Answer 1

3
$\begingroup$

I'm going to introduce some of the subjects that you require to know before you can understand deep learning. My list is not exhaustive, of course. But it should cover more than $60\%$ of the math you need to know. I repeat again that this list is supposed to be minimal and once you start learning these topics, you will realize that you need to understand a bit of other things too. But these ones are the core subjects that you should know.

Calculus (estimated time: 3 months - 6 months)

You need to know calculus well. Most importantly, you need to understand multivariable calculus well because in most problems in machine learning (and neural networks), we're dealing with functions that have many inputs. So, I would say that you need to spend a good amount of time learning about scalar functions, vector functions, gradient, the Hessian matrix, potential functions, Lagrange multipliers, Laplacian, line integrals, Stokes' theorems, etc.

Probability theory (estimated time: 3 months - 6 months)

Computers are wired to deal with true or false values. However, humans think differently. The thing that enables us to teach computers to imitate how we make decisions is probability theory. You need a basic course in probability theory. At the end, you should understand things like a probability space, an event, probability mass/density functions, conditional probability, expectation, variance, higher order moments, characteristic function, the central limit theorem, Bayesian inference, etc. The fun starts after you understand Bayesian thinking. But that's just the beginning. Then you should try to understand harder stuff like Markov chains, stochastic processes, MCMC methods, etc.

Linear algebra (estimated time: 3 months - 6 months)

Linear algebra is probably the most powerful subject in mathematics. You can never learn enough linear algebra. In most cases, dealing with functions is difficult. Nonlinear functions are complicated and they can have very weird behavior in general. However, some functions (which turns out to be almost all of the functions that we're interested in) happen to be well approximated by linear functions. This approximation is done in calculus. The derivative of a function is its linear approximation. Also, linear algebra helps us convert our problem to a language that can be implemented easily in computers. Computers know how to manipulate numbers, vectors, matrices or algebraic objects in general. After you study linear algebra, you should be able to understand the following concepts: systems of linear equations, Gaussian elimination, vector spaces, linear independence, spanning sets, bases, linear transformations, kernel, range, rank-nullity theorem, determinants, eigenvalues and eigenvectors, inner product spaces, canonical forms like diagonalization, Jordan normal form, etc.

Mathematical Analysis (estimated time: 6 months - 9 months)

Mathematical analysis is something that combines your understanding of most of the topics from the subjects above. It helps you understand calculus better. It helps you understand probability theory from a rigorous point of view. And it helps you understand why many of our numerical methods work. After you study mathematical analysis, you should be able to understand the following topics: the field of real numbers, open sets, closed sets, limit points, convergence, compactness, connectedness, completeness, metric spaces, differentiation, integration, Taylor's theorem, functional spaces, Jacobian matrix, Riemann integration, a bit of measure theory, etc.

$\endgroup$
2
  • 2
    $\begingroup$ Thank you very much for this good overview! I appreciate the time you took for writing this! $\endgroup$ Jan 27, 2019 at 15:25
  • $\begingroup$ @postnubilaphoebus You're welcome. I wish you the best in your journey for understanding deep learning. $\endgroup$ Jan 27, 2019 at 15:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .