24
$\begingroup$

I've been thinking about this question already for a long time and I've just encountered it again in the following lemma:

$$f(x) = g(Ax + b) \implies \nabla f = A^T \nabla g(Ax + b) $$

This lemma makes intuitive sense if you think of it as taking the $x$ to the space $Ax$, calculating the gradient and then taking the result back to the original space. But why is "taking the result back" realised as $A^T$ and not $A^{-1}$?

By doing the calculations you get $A^T$, no doubt, but I always expect an inverse. In general, when should I expect a transpose and when an inverse? Where are they similar and where do they differ?

$\endgroup$
2
  • 11
    $\begingroup$ Think about the 1-dimensional case: if $f(x) = g(ax+b)$, then $f'(x) = g'(ax+b)a$, not $g'(ax+b)/a$. So it can't be the inverse; it must be the transpose. $\endgroup$ May 15, 2018 at 18:47
  • 3
    $\begingroup$ Also, the row matrix $\mathrm{D}f = (\nabla{f})^\top$ is more fundamental than the column matrix $\nabla{f}$. So you should be thinking $\mathrm{D}f(x) = \mathrm{D}g(Ax+b)A$ to begin with; then you can explicitly take the transpose of each side, if you must, to get $\nabla{f}(x) = A^\top\nabla{g}(Ax+b)$. But fundamentally, it's not about the transpose or the inverse. $\endgroup$ May 15, 2018 at 18:52

5 Answers 5

19
$\begingroup$

We usually see matrices as linear transformations. The inverse of $A$, when it exists, means simply "reversing" what $A$ does as a function. The transpose originates in a different point of view.

So we have vector spaces $X,Y$, and $A:X\to Y$ is linear. For many reasons, we often look at the linear functionals on the space; that way we get the dual $$ X^*=\{f:X\to\mathbb R:\ f\ \text{ is linear}\}, $$ and correspondingly $Y^*$. Now the map $A$ induces a natural map $A^*:Y^*\to X^*$, by $$ (A^*g)(x)=g(Ax). $$ In the particular case where $X=\mathbb R^n$, $Y=\mathbb R^m$, one can check that $X^*=X$ and $Y^*=Y$, in the sense that all linear functionals $f:\mathbb R^n\to\mathbb R$ are of the form $f(x)=y^Tx$ for some fixed $y\in\mathbb R^n$. In this situation $A$ is an $m\times n$ matrix, and the matrix of $A^*$ is the transpose of $A$.

$\endgroup$
3
  • 3
    $\begingroup$ You're the only one who answered my question. I wasn't asking about how to derive the formula, but everyone tried to derive it for me. You actually gave me insight into where the transpose comes from. Thank you $\endgroup$
    – LionCoder
    May 15, 2018 at 23:25
  • 2
    $\begingroup$ Glad I could help. The whole thing is more visible in more abstract spaces than $\mathbb R^n$. In particular the adjoint plays a big role when dealing with HIlbert spaces and their operators. And the star in "C$^*$-algebra", for instance, comes from the notation for the adjoint. $\endgroup$ May 15, 2018 at 23:28
  • 1
    $\begingroup$ Just to add a little bit, you can then treat $\nabla f(x)$ as a linear function, whose value $[\nabla f(x)](y)$ at $y$ is the dot product of $y$ and the gradient. In particular, the gradient is the function for which $g(x) = f(x_0) + [\nabla f(x_0)](x)$ is the best affine approximation of $f$ at $x_0$. Then $f(Ax_0) + [\nabla f(Ax_0)](Ax) = f(Ax_0) + [A^* \nabla f(Ax_0)](x)$ is the best linear approximation of $f(Ax)$ at $x_0$. The formula in the question is just the matrix representation of this. $\endgroup$ May 16, 2018 at 13:20
10
$\begingroup$

Something weird is going on here. I'm assuming $g: \mathbb R^m \to \mathbb R$ and say $A$ is an $m\times n$ matrix. Let $\mathcal a(x): \mathbb R^n \to \mathbb R^m, x \mapsto Ax + b$ be the corresponding affine transformation, so that $f = g \circ a$. The chain rule says $Df(x) = Dg(a(x)) Da(x)$.

The Jacobian realization of $Dg$ is $\nabla g$ and is an $1\times m$ matrix (row vector), while the Jacobian for $a$ is $A$, an $m \times n $ matrix. The dimensions all agree, since this would make $\nabla f$ a $1\times n$ matrix, which agrees with the notion that the derivative of $f$ is a linear map $\mathbb R^n \to \mathbb R$.

So what I suspect is happening is some identification of $\mathbb R^n$ with its dual space under the Euclidean inner product; that is, you're realizing the gradient as a column vector instead of a row vector. The transpose is precisely the way this is done. If $T: V \to W$ is a linear transformation, then its adjoint is $T^\dagger: W^* \to V^*$. But under the Euclidean inner product, you can identify $\mathbb R^n \cong (\mathbb R^n)^*$, so $$ (\nabla g(a(x)) A)^T = A^T [\nabla g(a(x))]^T = A^T \nabla g(a(x))$$ where we're abusing notation by identifying the row vector $\nabla g$ with the column vector $\nabla g$. This hidden identification is likely what is confusing you.

$\endgroup$
6
  • 7
    $\begingroup$ I really wish that we could just teach that the gradient is a row vector in multivariable calculus from the beginning. This confusion is just not necessary, and exacerbates the process of learning about the Jacobian (which is basically a "column vector of gradients"). I understand why this isn't practical, though. $\endgroup$
    – Ian
    May 15, 2018 at 15:38
  • $\begingroup$ @Ian I got yelled at once by a very senior faculty member for defining the derivative of the map $g: \mathbb R^n \to \mathbb R$ as row vector, rather than distinguishing the linear transformation from its Jacobian representation. This was in a non-advanced course :S $\endgroup$
    – S.Micheals
    May 15, 2018 at 15:43
  • $\begingroup$ @Ian Just curious. Why isn't that practical? $\endgroup$
    – user1551
    May 15, 2018 at 22:46
  • $\begingroup$ @user1551 There's no way to require all multivariable students to have enough familiarity with linear algebra to really understand the difference between row vectors and column vectors. $\endgroup$
    – Ian
    May 16, 2018 at 1:23
  • 3
    $\begingroup$ @Ian I wish we could stop talking about row or column vectors entirely, and instead start from abstract vector spaces and their dual spaces. $\endgroup$ May 16, 2018 at 12:00
8
$\begingroup$

Notice using the chain rule that $$D_p g(Av+b)=\langle\nabla g(Ap+b),Av\rangle=\langle A^T\nabla g(Ap+b),v\rangle.$$ Now compare to $D_pf(v)=\langle\nabla f(p),v\rangle$.

$\endgroup$
4
$\begingroup$

Here you are not "taking the result back to the original space", you are chaining transforms.

If you think of a linear transform applied to a vector, it's a bunch of dot products, of the rows of the array by the column vector and

$$\vec x\cdot\vec y\equiv x^Ty.$$

$\endgroup$
4
$\begingroup$

Taking the directional derivative of $f (\mathrm x) := g (\mathrm A \mathrm x + \mathrm b)$ in the direction of $\rm v$ at $\rm x$,

$$\lim_{h \to 0} \frac{f (\mathrm x + h \mathrm v) - f (\mathrm x)}{h} = \langle \nabla g (\mathrm A \mathrm x + \mathrm b), \mathrm A \mathrm v \rangle = \langle \mathrm A \mathrm v, \nabla g (\mathrm A \mathrm x + \mathrm b) \rangle = \langle \mathrm v, \mathrm A^\top \nabla g (\mathrm A \mathrm x + \mathrm b) \rangle$$

and, thus, the gradient of $f$ is

$$\nabla f (\mathrm x) = \mathrm A^\top \nabla g (\mathrm A \mathrm x + \mathrm b)$$

$\endgroup$
2
  • $\begingroup$ Essentially my answer given two hours before ... $\endgroup$ May 15, 2018 at 20:54
  • $\begingroup$ @MichaelHoppe When I wrote my answer, yours had not yet been posted. You can accuse me of not checking other answers before posting mine, but you cannot accuse me of plagiarism. $\endgroup$ May 16, 2018 at 4:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.