Could someone provide a good reference for learning matrix calculus? I've recently moved to a more engineering-oriented field where it's commonly used and don't have much experience with it.
Actually the books cited above by Sivaram are excellent for numerical stuff. If you want "matrix calculus" then the following books might be helpful:
Matrix Differential Calculus with Applications in Statistics and Econometrics by Magnus and Neudecker
Functions of Matrices by N. Higham
Calculus on Manifolds by Spivak
Some classic, but very useful material can also be found in
- Introduction to Matrix Analysis by Bellman.
As a simple example, the books will teach (unless you already know it) how to compute, say, the derivative of $f(X) = \log\det(X)$ for an invertible matrix $X$.
Gene Howard Golub, Charles F. Van Loan book on "Matrix Computations" is regarded as the "Bhagavad Gita" for Matrix Algorithms.
There is also another book by "Gene Howard Golub, Gerard Meurant" on "Matrices, moments, and quadrature with applications".
Also, "Numerical Linear Algebra" by Trefethen and Bau is well-written and easy to read.
I would highly recommend Trefethen and Bau since I have read it completely. I feel it is ideal for self-study or for a one quarter course. Once you are done with this you can take a look at Golubs' book. Golubs' book is really good for reference.