I am learning gradients on my own.

One of the rules given in the textbook I am reading states this (without proof):

$$∇_\mathbf{x}\mathbf{A}\mathbf{x} = \mathbf{A}^⊤$$

and then proceeds onwards as if it's trivial to see. I haven't seen this before so I need to go prove that it is correct.

I do not know if the above equation is equal to

$$∇f(\mathbf{x}) = \mathbf{A}^⊤$$


$$f(\mathbf{x}) = \mathbf{A}\mathbf{x}$$

I decided to go ahead with this anyways and see where it takes me. I figure doing this with a simple $\mathbb{R}^{2 \times 2}$ like $\mathbf{A} = \begin{bmatrix}a&b\\c&d\end{bmatrix}$ and seeing where it takes me when $\mathbf{x} = \begin{bmatrix}i\\j\end{bmatrix}$.

This means $\mathbf{A}\mathbf{x}$ is:

$$\begin{bmatrix}ai + bj\\ci + dj\end{bmatrix}$$

The gradient would be (for $\mathbf{Ax}$ as I'm passing that in for the parameter):

$$∇f(\mathbf{Ax}) = \begin{bmatrix} \frac{\partial f(\mathbf{Ax})}{\partial i} \frac{\partial f(\mathbf{Ax})}{\partial j}\end{bmatrix}^⊤$$

Which means we want to evaluate $∇_\mathbf{x} \begin{bmatrix}ai + bj\\ci + dj\end{bmatrix}$. By the definition of a gradient, then we should get $∇f(\mathbf{Ax}) = \begin{bmatrix} \begin{bmatrix}a\\c\end{bmatrix} \\ \begin{bmatrix}b\\d\end{bmatrix} \end{bmatrix}$

This however does not look right, or is it? Is that equal to $\begin{bmatrix}a&c\\b&d\end{bmatrix}$?

I wrote it out that way because I suspect that when we ask for $a_{21}$ we are saying "get the 2nd row in the first column, and in a programming fashion, A[2][1] (or A[1][0] for most languages) would get $b$ from the result above. This would make sense because I'd get the transpose I was looking for in the primitive example. My linear algebra class only covered vectors, then jumped to matrices and showed us matrix multiplication without covering matrices as if they were a vector of vectors, so if there is a fundamental gap in my knowledge here and what I've described above is correct... then we've found an issue.

Moving to an n-dimensional proof after figuring out what is wrong (if anything) seems like it'd be straight forward with more rigorous variable and index naming.

  • 1
    $\begingroup$ The transpose should not be there, at least if you are using most conventional definitions of the gradient. What book are you using, and what is the definition of the gradient in your book? $\endgroup$
    – Nick Alger
    Dec 30 '20 at 3:20
  • $\begingroup$ If $f: \mathbb R^n \to \mathbb R^m$ is differentiable at $x$, then $f'(x)$ is a matrix such that $f(x + \Delta x) \approx f(x) + f'(x) \Delta x$. The approximation is good when the vector $\Delta x \in \mathbb R^n$ is small. This approximation, which is sometimes called Newton's approximation, is the best way to think about the derivative of a function $f: \mathbb R^n \to \mathbb R^m$. Now, if it so happens that $f(x) = Ax$ for some matrix $A$, then $f(x + \Delta x) = Ax + A \Delta x = f(x) + A \Delta x$. Comparing this with Newton's approximation reveals (or suggests) that $f'(x) = A$. $\endgroup$
    – littleO
    Dec 30 '20 at 3:23
  • $\begingroup$ @NickAlger most likely this is a book on machine learning or something similar. The convention in that field is that gradients are transposes of what they reasonably should be e.g. gradients of scalars being column instead of row vectors. $\endgroup$ Dec 30 '20 at 3:24
  • $\begingroup$ In a context where we're doing gradient descent, it's not crazy to want the gradient of a function $f: \mathbb R^n \to \mathbb R$ to be a column vector, so that the iteration $x^{k+1} = x^k - t \nabla f(x^k)$ makes sense. So in my mind (coming from an optimization background), if $f: \mathbb R^n \to \mathbb R$ is differentiable at $x$, then $f'(x)$ is a $1 \times n$ row vector, and $\nabla f(x) = f'(x)^T$ is a column vector. (However, I would never refer to the "gradient" of a function $f: \mathbb R^n \to \mathbb R^m$.) $\endgroup$
    – littleO
    Dec 30 '20 at 3:55
  • $\begingroup$ @NickAlger This is the one I'm using: d2l.ai/chapter_preliminaries/calculus.html#gradients $\endgroup$
    – Water
    Dec 30 '20 at 13:32

Suppose that your matrix A is $m \times n$. First of all, you need to view $Ax$ as a function $f: \mathbb{R}^{n} \mapsto \mathbb{R}^{m}$ such that \begin{equation} \forall x \in \mathbb{R}^{n}, f\left(x\right) = Ax. \end{equation} Basically, this function is a linear transformation of $\mathbb{R}^{n}$. Then you should understand how derivative is defined for functions from multiple inputs to multiple outputs. The definitions and theorems can typically be found in real analysis. A function is differentiable at $x_{0}$ if and only if there exists a matrix $M$ such that \begin{equation} \lim_{x \to x_{0}}\frac{\lVert f\left(x\right)-f\left(x_{0}\right)-M\left(x-x_{0}\right)\rVert}{\lVert x - x_{0}\rVert} = 0. \end{equation} Then we write \begin{equation} M = f^{\prime}\left(x_{0}\right). \end{equation}

Here, you may understand $\nabla$ as the concept differential $D$. The derivative of $f$ at a point $x \in \mathbb{R}^{n}$ is described as follows: \begin{equation} f^{\prime}\left(x\right) = \begin{pmatrix} \frac{\partial f_{1}\left(x\right)}{\partial x_{1}} & \frac{\partial f_{1}\left(x\right)}{\partial x_{2}} & \dots & \frac{\partial f_{1}\left(x\right)}{\partial x_{n}}\\ \frac{\partial f_{2}\left(x\right)}{\partial x_{1}} & \frac{\partial f_{2}\left(x\right)}{\partial x_{2}} & \dots & \frac{\partial f_{2}\left(x\right)}{\partial x_{n}}\\ \vdots & \vdots & \ddots & \vdots\\ \frac{\partial f_{m}\left(x\right)}{\partial x_{1}} & \frac{\partial f_{m}\left(x\right)}{\partial x_{2}} & \dots & \frac{\partial f_{m}\left(x\right)}{\partial x_{n}} \end{pmatrix}, \end{equation} which is a $m \times n$ matrix. Sometimes, the derivative is defined with a transpose symbol, which is essentially your case. The differential at a point $x_{0}$ is of the following property: \begin{equation} D_{x_{0}}f\left(x\right) = f^{\prime}\left(x_{0}\right)x, x \in \mathbb{R}^{n}. \end{equation} That is to say, the differential is a function $D_{x_{0}}f: \mathbb{R}^{n} \mapsto \mathbb{R}^{m}$. Again, real analysis is the best reference for these questions. An open-source book I recommend is "Introduction to Real Analysis" by William F. Trench.


$\def\p#1#2{\frac{\partial #1}{\partial #2}}\def\d{\delta}$In index notation, the calculation is clearly $$\eqalign{ y_i &= A_{ij}x_j \\ dy_i &= A_{ij}dx_j \\ \p{y_i}{x_k} &= A_{ij}\left(\p{x_j}{x_k}\right) = A_{ij}\d_{jk} = A_{ik} \\ }$$ but there is an ambiguity in how the indices on the LHS should be ordered, i.e. is the gradient $G_{ik} \;{\rm or}\; G_{ki}$?

This really depends on your approach to computing the differential $$\eqalign{ dy &= \; (G\cdot dx) \;\overset{\;?}{\iff}\; (dx\cdot G) \\ }$$ I have a strong preference for the first form, but opinions vary.

The ambiguity extends to higher-order tensors as well. $$\eqalign{ \p{Y_{ij}}{X_{k\ell m}} &=\;\; \Gamma_{ijk\ell m} \;\overset{\;?}{\iff}\; \Gamma_{k\ell mij} \\\\ }$$ These ideas also carry over to one's preference for integral expressions, e.g. $$\eqalign{ \int G\cdot dx \quad\overset{\;?}{\iff}\quad \int dx\cdot G \\ }$$ Again, I prefer the first form.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.