# How should Matrix Calculus be thought of with respect to Vector and Multi-variable Calculus?

My question boils down to identifying the ways in which Matrix Calculus is distinct from Vector and Multi-variable Calculus, and how they overlap. Is Matrix Calculus simply a notation on top of these other types of Calculus, or does it actually contain new theorems/lemmas/results separate from the other two? The subject of Matrix Calculus does not seem to be a subject of courses at universities and I struggle to find resources to actually learn about it.

In mathematics, matrix calculus is a specialized notation for doing multivariable calculus, especially over spaces of matrices... The notation used here is commonly used in statistics and engineering, while the tensor index notation is preferred in physics. (Emphasis mine.)

To answer your question directly, I would says that it is "simply a notation on top of these other types of Calculus".

To answer your question with hopefully useful additional context, I think the distinction between "multivariable calculus" and "vector calculus" can often be more philosophical then anything else. For instance, given a function $$f(x,y) = xy+y^3$$, is this a function defined on two real variables $$x$$ and $$y$$ or defined on the vector $$\mathbf{x} = (x,y)$$? In multivariable calculus, as taught in standard undergraduate courses, we often adopt the former approach. In this setting, the natural derivatives are the partial derivatives $$\partial f/\partial x$$ and $$\partial f/\partial y$$. In the vector calculus framework, it is more natural to consider the derivative of $$f$$ to be the gradient $$\nabla f$$, which is a vector$${}^*$$ whose components are $$\nabla f = (\partial f/\partial x, \partial f/\partial y)$$. But note that whether we think of the gradient or the partial derivatives, the computations are ultimately the same.

Particularly in three dimensions, the vector calculus framework becomes very helpful to adopt as one progresses into multivariable calculus because the fundamental integration theorems of multivariable/vector calculus (namely the divergence theorem and Stoke's theorem) are most understandable when phrased in terms of vector operations like the dot and cross products. But in principle you can just do the conversion to partial derivatives and all of the "vectors" totally disappear.

This brings us to matrix calculus. Suppose we have a function of four variables $$f(x,y,z,w)$$. We could consider this a function of four real variables, one four-dimensional real vector $$\mathbf{x} = (x,y,z,w)$$, or the $$2\times 2$$ matrix $$X = \begin{pmatrix} x & y \\ z & w \end{pmatrix}$$. In the end, no matter which viewpoint we adopt, the calculations all boil down to partial derivatives and multiple integrals. This is why there are no dedicated courses on matrix calculus--there's nothing really new to the game.

On Wikipedia you can scroll through seemingly endless formulas with derivatives of scalars with respect to matrices and vectors with respect to other vectors, and it can seem like a lot. If you have a function that takes $$n$$ numbers $$x_1,\ldots,x_n$$ (be they arranged as a vector or matrix or treated as individual elements) and spits out $$m$$ numbers $$y_1,\ldots,y_m$$, then there a total of $$n\times m$$ partial derivatives $$\partial y_1/\partial x_1, \partial y_2/\partial x_1,\ldots$$ and all the differential and integral identities boil down to the same calculations you did in multivariable calculus (a lot of chain rule, product rule, etc.)

Often, it is convient to arrange thse $$n\times m$$ numbers into a special format as a vector or matrix and give it a name. For instance, given a function taking in $$n$$ real numbers and spitting out $$m$$ real numbers, we often arrange them into an $$m\times n$$ Jacobian matrix. The most common ways of arranging the outputs of these calculuations are listed on the Wikipedia page I referenced earlier.

So fear not, for many problems in the natural sciences and machine learning, a strong understanding of traditional undergraduate multivariable calculus (sprinkled with a healthy understanding of linear algebra) should be plenty. When you see some fancy formula involving a matrix derivative or the like, just ask yourself how the partial derivatives are arranged into a matrix. (There are often multiple competing definitions.) Often, since these things are context dependent, the author will often give you the answer in a notation section or the like.

There is indeed are much more deep ways to think about multivariable calculus, largely housed in tensor calculus and the like. There are many good books at many different levels on these subjects, and are the subject of many university courses (usually as part of differential geometry).

$${}^*$$ More precisely, the gradient is better thought of as a linear functional or co-vector. In matrix language, it's better to think of the gradient as a row vector than a column vector.

• Wow! This was exactly the missing contextual link I was looking for and is very well written! To make sure I understand, is it a correct summary to say that multivariable calculus contains the base material and the "vector" and "matrix" flavors are simply notations which make use of linear algebra to neatly package and manage that material when dealing with higher dimensions than 2? Is it correct to say that tensor calculus is a generalization of the concepts of multivariable calculus in the same way that multivariable calculus is a generalization of single variable calculus? – Adam Sperry Jun 6 '19 at 15:17
• @Adam Sperry To answer your first question, yes. (Although, it might be equally valid to say that multivariable calculus is just a notation on top of vector calculus.) Everything in vector calculus works with dimensions beyond three except cross products, which only work in dimension three (and kind of two). This is where tensor calculus jumps in, giving a version of Stokes theorem that works in any number of dimensions. The well of tensor calculus (and its relatives) goes very very deep, most of it is beyond me at my point in my mathematical journey. – eepperly16 Jun 6 '19 at 19:33
• Thank you so much! – Adam Sperry Jun 6 '19 at 19:53