3
$\begingroup$

I am taking a course in machine learning and have found that the linear algebra and multivariable calculus from my engineering degree only take me part way in understanding some derivations.

One specific example is differentiating things related to matrices (like differentiating wrt a matrix whose determinant appears in the function...) But this is by no means the only fuzzy bit. I have done just enough geometry, linear algebra and low dimensional calculus to have a notion that these things somehow extend into things involving matrices and things involving high dimensional spaces.

I've tried looking on places like wikipedia's topic listing in mathematics, the page "Categories within Mathematics" at arxiv.org, and undergraduate mathematics curricula however I don't think I even know enough to know whether I'm looking at what I'm looking for... if that makes sense....

Also, I've found some compilations of matrix derivatives but a) it's disconnected from any context and b) seems like a cookbook solution and so these haven't been satisfying.

So... how does what I'm saying here map to topics in mathematics?

If I can cheat and ask a "sub question"... whatever these topics end up being, what are some typical paths people take to get from fairly applied linear algebra and calculus to these "advanced" topics? Items on such a path could be books, names of courses, whatever... I just need some guidance towards context and prerequisites.

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ Differentiating things related to matrices is a topic which one needs in many applied areas, but which I haven't seen taught systematically in any course. It seems to be called matrix calculus, and J. Dattorro has a book chapter about it online. Personally, I usually just derive the results I need via differentials: to differentiate a function of a matrix $f(A)$, expand $f(A+dA)$ and ignore higher-order terms in $dA$. $\endgroup$ – Rahul Oct 4 '18 at 16:22
  • $\begingroup$ Thanks @Rahul. If you've got any ideas on the other aspects of the question, post an answer :) $\endgroup$ – Paul Oct 4 '18 at 18:04
  • $\begingroup$ To be honest I have no idea what the other aspects of the question are. I know you said differentiating functions of matrices is "by no means the only fuzzy bit", but you haven't said what the other bits are fuzzy, only that you want to learn about "whatever these topics end up being". So I answered about the only specific thing I could see in the question. $\endgroup$ – Rahul Oct 4 '18 at 18:16
  • $\begingroup$ @Rahul Fair enough, I'm definitely having trouble with this. I think calculus, geometry, and probability/stats in high dimensional spaces definitely is part of what I'm looking for- somehow linear algebra and matrices fit in (maybe that sounds obvious), and I've also come across the term "tensor" in some of my searching, I think maybe in relation to the covariance matrix? $\endgroup$ – Paul Oct 4 '18 at 19:27
3
+50
$\begingroup$

I am an engineer, and need to use the "practical" side of these things on a daily basis. Assuming you have a strong grasp of 3D calculus and linear algebra, you probably want to start with a bit of each of the following:

  • Multilinear algebra. This is where, e.g. tensors appear. Key topics are:

  • Calculus on (finite-dimensional, real) Banach spaces, generalizing (some of) what you learned in multivariable calc. Key topics are:

    • Directional (Gateaux) derivatives. Understand the isomorphism between vectors and directional derivatives along vectors ($v \sim (f \mapsto \nabla_vf$)).
    • Total (Frechet) derivative. The usual $f'$ is a special case of this, although you'll need to start thinking of $f'$ as a linear map $v \mapsto f'(a)v$ rather than a "number".
    • The Jacobian matrix as the matrix representing the full derivative with respect to standard coordinates.
    • Understand higher-order full derivatives as multilinear forms, at least up to the second derivative. For scalar valued functions, understand how the Hessian matrix is the matrix representation of the second (full) derivative.
    • Integration of vector-valued functions on Euclidean space. I'm not sure there's really more to learn here than what you learned in multivariable calc.
    • Integration of differential forms on chains, if you're feeling extra brave. Understanding the exterior algebra is a prerequisite to this.

These should at least get you more comfortable working with calculus on more sophisticated objects than functions from $\mathbb R^n \to \mathbb R$. Once you're there, you may be interested in doing calculus on non-linear spaces... e.g. how do you differentiate a function on a matrix group, or on the sphere? At that point you may want to dip your toes into differential geometry... Key topics would include pushforwards of maps between manifolds, integration of differential forms, affine connections, Lie groups and the exponential map, etc... But that's probably a ways away for now :)

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks Adam. Looks like you've earned the bounty. I see the linked Wikipedia articles have references... are these going to be accessible relative to my background? What kind of courses did you take as an engineering student? $\endgroup$ – Paul Oct 12 '18 at 1:37
  • $\begingroup$ Hmm, the first question is a bit difficult to answer, since (a) I'm not generally familiar with the references linked on the Wikipedia pages and (b) I don't really know your background. Re: the second question, my education was in math, which limits my ability to recommend specific references for people who are looking for engineering or application-based treatments. As maybe a place to start, Spivak's Calculus on Manfolds gives a thorough treatment of pretty much everything on my list, but I can't be sure whether it's the most appropriate source for your background and goals. $\endgroup$ – Adam Williams Oct 12 '18 at 23:31
  • $\begingroup$ Also it's worth noting that I'm essentially advocating a "top-down" learning approach here, wherein you understand the theory well enough that concrete calculations become intuitive, you can check your work based on first principles. This approach (as with any learning method) may not be best for everyone)! $\endgroup$ – Adam Williams Oct 13 '18 at 0:07
  • $\begingroup$ Thanks for clarifying... I misinterpreted when I read you say you're an engineer. I would ideally like to go for the "top down" approach you mentioned. If this book by Spivak is central to what you're talking about, my questions collapse down to a very simple one: if you take a first year university student, what preparation do they need to successfully tackle this book? $\endgroup$ – Paul Oct 14 '18 at 0:47
  • $\begingroup$ From the preface: The formal prerequisites include only a term of linear algebra, a nodding acquaintance with the notation of set theory, and a respectable first-year calculus course (one which at least mentions the least upper bound (sup) and greatest lower bound (inf) of a set of real numbers) . Beyond this a certain (perhaps latent) rapport with abstract mathematics will be found almost essential. $\endgroup$ – Adam Williams Oct 19 '18 at 5:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.