# What is the significance of theoretical linear algebra in machine learning/computer vision research?

I am a computer science research student working in application of Machine Learning to solve Computer Vision problems.

Since, lot of linear algebra(eigenvalues, SVD etc.) comes up when reading Machine Learning/Vision literature, I decided to take a linear algebra course this semester.

Much to my surprise, the course didn't look at all like Gilbert Strang's Applied Linear algebra(on OCW) I had started taking earlier. The course textbook is Linear Algebra by Hoffman and Kunze. We started with concepts of Abstract algebra like groups, fields, rings, isomorphism, quotient groups etc. And then moved on to study "theoretical" linear algebra over finite fields, where we cover proofs for important theorms/lemmas in the following topics:

Vector spaces, linear span, linear independence, existence of basis. Linear transformations. Solutions of linear equations, row reduced echelon form, complete echelon form,rank. Minimal polynomial of a linear transformation. Jordan canonical form. Determinants. Characteristic polynomial, eigenvalues and eigenvectors. Inner product space. Gram Schmidt orthogonalization. Unitary and Hermitian transformations. Diagonalization of Hermitian transformations.

I wanted to understand if there is any significance/application of understanding these proofs in machine learning/computer vision research or should I be better off focusing on the applied Linear Algebra?

-
I would recommend sticking to Applied form. Engineering in most form doesn't require the kind of rigor provided by H&K or for that matter, any mathematical subject. You need rigor only if you want to go about proving things or reading papers. – Inquest Feb 24 '12 at 8:46
@Nunoxic I am sorry to say that not everybody who does Math rigorously do it for going about proving things or reading papers. It is mostly fun and widens the scope for creative thoughts. – user21436 Feb 24 '12 at 9:03
I'm maybe wrong, but I think Theoretical Computer Science would be a better place for this question, isn't it? – Pascal Qyy Feb 24 '12 at 9:08
@KannappanSampath, I agree, it does influence the way one thinks but considering the opportunity cost of learning H&K, I doubt its worth taking a course. Maybe, it is worth looking into later (once you already know LA). Plus, something I have realized is, for engineers, the first course should ideally not be a rigorous course. It should be applied and later, if you are interested, pick up a concise rigorous text. But that's just my opinion. – Inquest Feb 24 '12 at 9:19
@stressed_geek: If you are researching Machine Learning and Computer Vision, presumably you are enrolled in some program and have several teachers and at least one adviser? "What kind of linear algebra -- e.g., which textbook -- would be most useful to me for my work?" is an eminently reasonable question to ask any of these people. And you should ask them: they will know much better than any of us, because they will know you and what you are working on much better than we do. – Pete L. Clark Feb 24 '12 at 19:49

If you want to do advanced computer vision, and not just implement algorithms, you will need to understand advanced algebraic concepts for linear transformations. You will also need to understand a bit of measure theory and analysis.

Why?

Because research level computer vision involves the development of algorithms. The development of these algorithms necessarily invokes the structural properties of the mathematical objects; properties such as measure, convergence, isometry, isomorphism, etc.

Furthermore, say you have the mechanical skills to develop a computational method. Any true research-level effort is also expected to demonstrate a proof of convergence, establish a domain in which the method is efficacious, compare the method to prior methods, and fundamentally compare the weaknesses and benefits.

This requires at least a solid understanding of graduate-level analysis and linear algebra.

-

From my personal experience, i think the most important topics are Probability, Statistics and Matrix Algebra. Of course, the basics of Linear Algebra are also required, but i guess that goes without saying.

The topics which you mentioned in the course of linear algebra, they can be good to know. For example, there are many unsolved problems in current methods of machine learning. If you have strong foundation of linear algebra, may be you can come up with solution to such existing problems.

Hope this helps.

-

Personally, linear algebra will be very important for your research in machine learning. Because it provides not only the best interpretaion of many real problems, but also gives the easy solutions to these problems, such as linear regression model and linear classification model. In addition, with the knowledge of linear algebra, it will be easier to study other courses including statistics which is important for machine learning.

Maybe, I can recommend a book for you on linear algebra, "Linear Algebra Done Right" edited by Sheldon Axler.

Best!

-