In single variable calculus, the chain, product, and power rules are very straightforward. In vector calculus, it's a bit more involved.

I frequently find myself having to do a lot of work to arrive at the answer. For example, taking the derivative of

$$ \frac{\partial}{\partial x} Ax $$

where $x$ is a column vector and $A$ is a matrix, I usually end up doing the following, where I find the i-th row of the expression and then take its derivative and generalize that to the full case

$$ (Ax)_i = \sum_{j}A_{ij}x_j \\ \frac{\partial}{\partial x_k}(\sum_j A_{ij}x_j) = A_{ik} \\ \therefore \frac{\partial}{\partial x} = A^T $$

But in this case, we could have just used the analogous power rule, and take the constant coefficient of $Ax$ (i.e., $A^T$, when using a denominator layout) to be the derivative.

This is an easy case. Now what if we had $$ \frac{\partial}{\partial x}x^TA $$

In this case, it's not easy for me to identify what the derivative should be, so I write out the terms individually

$$ x^TA = \left[\sum_{i}x_iA_{i1} \ \ \sum_{i}x_iA_{i2} \ldots \right] \\ \frac{\partial}{\partial x_k} x^TA = A_k \\ \therefore \frac{\partial}{\partial x} x^TA = A $$ Same as the derivative of $Ax$.

Now we can get more complicated, by having expressions like $x^TAx$, or taking higher order derivatives, where it can become tedious to write out the terms as I've done. To avoid making the post too long, I'll just briefly show how I would compute the derivative of $x^TAx$.

\begin{align} x^TAx = \langle x, Ax \rangle = \langle Ax, x \rangle \\ = \sum_i x_i \sum_j A_{ij}x_j \\ \frac{\partial}{\partial x_k}\sum_i x_i \sum_j A_{ij}x_j = \sum_j A_{kj}x_j + \sum_i x_i A_{ik} \ \ \ \ \ \text{where I applied the product rule} \\ \therefore \frac{\partial }{\partial x}(x^TAx) = Ax + A^Tx \end{align}

So basically my question boils down to, is there a quick way or some trick to find the derivative without going through summations (i.e., writing out the terms individually)? The power rule seems identical to the single variable case, where $\frac{d}{dx} cx = c$ and analogously $\frac{d}{dx} Cx = C$. The product rule seems more involved, and I can't readily identify a generalization of the multivariable case.

Wikipedia has some tables for common derivatives, and those are more or less the ones that I come across, but I don't want to commit those to memory, and would rather know a good strategy to compute these fast.

  • $\begingroup$ Have you read through The Matrix Cookbook? math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf $\endgroup$ – whpowell96 May 22 '20 at 23:43
  • 3
    $\begingroup$ You can find it invariantly by taking variations $F(x+dx)-F(x)$ and keeping only terms linear in $dx$. The "derivative" is the linear map acting on $dx$, i.e. a linear expression into which $dx$ is to be inserted. For example, $A(x+dx)-Ax=Adx$, $(x+dx)^TA-x^TA=dx^TA=(A^Tdx)^T$, $(x+dx)^TA(x+dx)-x^TAx=dx^TAx+x^TAdx+o(dx)$, etc. In particular, your "simplified" expressions for the "derivative" in the second and third case are incorrect. $\endgroup$ – Conifold May 23 '20 at 0:11
  • $\begingroup$ @whpowell96 I think I've been led to that link several times when I google'd for things, but never read the book as a whole. $\endgroup$ – anonuser01 May 23 '20 at 0:49
  • $\begingroup$ @Conifold. I'm using the denominator layout for the derivative shown in en.wikipedia.org/wiki/… rather than the numerator layout, so the last 2 cases should be correct, and the first case is wrong, and should be $A^T$. $\endgroup$ – anonuser01 May 23 '20 at 0:49
  • $\begingroup$ @Conifold The variations approach seems really nice. I've never heard of it. It seems based on a taylor series expression and keeping only the linear terms? Also, since $dx$ is a vector, how do you reorganize the RHS of the expression so there isn't a $dx$? $\endgroup$ – anonuser01 May 23 '20 at 0:56

This is such a cool problem that i had the change to learn recently! Let's first introduce what is a tensor. So a tensor is a n dimensional array of numbers. Very familiar examples: if $n=2$ then the tensor is a matrix. If $n=1$ then the tensor is a vector. So now that we know what is a tensor, we can introduce the notion of a tensor network.

Please take a look at some basics of a tensor network in the shared link. But very informally speaking, a tensor network is a graph representation of a products of tensors like in the example of taking a product between a matrix and a vector or in the example of taking the product of a matrix with a matrix. How this graph of tensor network represent the product between tensors is hopefully not very hard to understand. The tensor netowrk graph is a graph in which the vertices of the graph represent a tensor involed in the product and we have that two tensor are connected via a labelled edge if we are taking their product in the tensor product.

The reason that we talk about the tensor network here is really that it becomes easier when we want to perform the task of taking the partial derivatives of a tensor product with respect to the entries inside of a tensor involved in the product. So how is this being done? The operation of taking the partial derivatives of the tensor product with respect to the entries of a tensor is represented by the tensor network that you would get if you were to delete the tensor that you are taking the partial derivative of in the tensor network representing the tensor product.

we stick with the tensors being only matrices and vectors and see an example of how taking partial derivatives of product of matrices and vectors is done in the language of tensor product. A matrix can be represented as a node with two edges in a tensor network. The two edges represent the rows and columns of the matrix. For example:

If we have a random $3$ by $2$ matrix like this: $A=\begin{pmatrix}a & b\\\ c & d \\\ f &g\end{pmatrix}$ The matrix $A$ can be represented as the tensor network in the following picture that I made with a lot of effort:

enter image description here

Here, as you can see, it is a node with two free edges sticking out of the node. Here, the left edge of node $A$ represent the columns of matrix $A$ and the right edge of the node $A$ represent the rows of node $A$. The number on top of the edges are used to denote the dimensions of the rows and columns respectively. In this case, you see that the rows are in 2 dimensional spaces and the columns are in the $3$ dimensional spaces therefore the left edge has the number 3 on top and the right edge has the number $2$ on top.

We are then going to introduce two vectors $u \in \mathbf{R}^2, v \in \mathbf{R}^3$, and let us now take a look on how to represent the product $v^TAu$ in a tensor network that we have defined, with a lot of work, above. So we first draw our vectors in tensor network format:

enter image description here

We notice that the nodes $u,v$ represented as tensor network each have only one edge coming out of it. This makes sense as we said before that the number of edges that sticks out of a tensor network represents the dimension of the resulting tensor in the tensor network. And the vector are 1-dimentional so we have gotten the right representation of the two vectors.

We now create out tensor network for the matrix-vecor product $v^TAu$. The way we do it is that we just have to attach them corrected via their sticked out edges according to how you multiply the matrix with the vectors. For example, the vector $v^T$ is multiplying the columns of the matrix $A$ and their dimension matches, so we just have to connect the vector with the matrix together where the vector $v^T$ connect to the matrix $A$ via the left edge and the vector $u$ is connected to the matrix $A$ via the right edge.

enter image description here

So now we can notice the resulting tensor network has no edges that sticks out. Because again the number of edges that sticks out of a tensor network represent the dimension of the resulting tensor network, this means that the resulting tensor network representing the product $v^TAu$ is a scalar. This is indeed true as the result of the products $v^TAu$ is indeed a scalar when you multiply them out. Now let's see how we can take derivative of the tensor network with respect to either the vectors or the matrix $u,v,A$. The way we do is as we stated is to just delete the corresponding tensor in the tensor network for with we are taking the derivatives of and we will get the resulting tensor network as our solution.

Let's see an example of taking a derivative in action, Let's take the partial derivatives of $v^TAu$ with respect to each entries in $u$. If we do it using the normal way of computing the partial derivatives, we would 2 partial derivatives that we can organize into a vector of dimension $2$. That vector, after we have done the calculation, is exactly $v^TA$ and you represent it in tensor network as :

enter image description here

as you can see, the result is a tensor network is a tensor with a free edge of dimension $2$. This resulting tensor network represent exactly the product $v^TA$.

Now, let's see a slightly more complicated computation. Let's see how we can take the partial derivative of $v^TAu$ with respect to each entries in the matrix $A$. If we were to do it by computing the partial derivatives, we will get 6 partial derivatives that we can organize them into a $3$ by $2$ matrix. We should therefore expect that the result is a matrix of dimension $\mathbf{R}^{3 \times 2}$ after we have deleted the matrix $A$ in the tensor network. If we compute the partial derivatives, The resulting matrix is the outer product of the two vectors $u$ and $v$. In tensor network format we get:

enter image description here

Notice that this is tensor network is different from that of the second image because in the second image, we were thinking of each vectors $u,v$ as its own tensor network, But the above picture, we think of the tensor representation of both $u,v$ as together as one tensor with two edges sticking out. Because as you can see, there are two free edges representing a two dimensional matrix. Notice when we have disconnected components in the tensor network, it means that we are taking the outer product of them.

There are one more case that we need to consider and that is what happen, if a tensor appears multiple times in the tensor network and you are taking the partial derivatives of the tensor network with respect to that tensor. In that case, you can consider removing each of occurrence of that tensor one at the time and then add the result tensor network to get the derivatives.

Hopefully this is very helpful to compute the partial derivatives as it was helpful to me:)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.