1
$\begingroup$

Let $f:M \subset \mathbb{R}^2 \rightarrow N \subset \mathbb{R}^3$.

  • The function $f$ is a vector function.
  • Its differential $\mathrm{d}f \in \mathbb{R}^3$ represents the infinitesimal change in the function, where by $\mathrm{d}f$, I mean $\mathrm{d}f(x)$.
  • Its Jacobian (matrix) $J \in \mathbb{R}^{3 \times 2}$ maps vectors between tangent spaces $T_x M$ and $T_{f(x)} N$.

The relation between the two is $\mathrm{d}f = J dx$, where $\mathrm{d}x \in \mathbb{R}^2$.

However, if $f$ is considered a "mapping", then is the differential of the mapping $\mathrm{d}f$ equal to the Jacobian $J$?


From some of the answers, it seems that I took some things for granted (common knowledge or agreed by all). Moreover, there seems to be a confusion between differential, derivative, and their notation.

So first, let's agree that the differential (total derivative) and the derivative (Jacobian) are not the same thing:

Next, as per Wikipedia, let's agree on notation. Each of $f'(x)$, $D f(x)$, and $\frac{\mathrm{d} f}{\mathrm{d} x}$, and $J$ refers to the derivative. The notation $\mathrm{d}f$ is reserved to denote the differential.

Now, back to my question.

  • The derivative of $f$ is the Jacobian matrix $f'(x)=Df=J \in \mathbb{R}^{3 \times 2}$.

  • The differential of $f$ is the 3D vector $\mathrm{d}f = J \mathrm{d}x$.

For some reason, there are people who confusingly use the term "differential of a mapping" to refer to the derivative, as if they don't distinguish between the derivative and the differential:

My question is: What's up with that, and what am I missing?

Why is that important: for a long time, I wasn't clear about what exactly the differential is. It became an issue when I used matrix calculus to calculate the Hessian of a matrix function. The book Matrix Differential Calculus with Applications in Statistics and Econometrics cleared it all up for me. It properly and distinctively defines the Jacobian, gradient, Hessian, derivative, and differential. The distinction between the Jacobian and differential is crucial for the matrix function differentiation process and the identification of the Jacobian (e.g. the first identification table in the book).

At this point, I am mildly annoyed (with myself) that previously I wrote things (which are too late to fix now) and blindly (relying on previous work) used the term "differential of a mapping". So, currently, I either look for some justification for this misnomer or otherwise suggest to the community to reconsider it.


I tried to track down the culprit for this "weird fashion", and I went as far as the differential geometry bible. Looking at do Carmo, definition 1 in chapter 2 appendix, pg. 128 (pg. 127 in the first edition), the definition of $dF_p$ is fine (grammar aside): it's a linear map that is associated with each point in the domain.

But then, in example 10 (pg. 130), he uses the same notation to denote both Jacobian and differential. (This is probably what Ulrich meant by almost the same thing.) More specifically, he "applies it twice": once to get the Jacobian and once to get the differential. He uses $df(\cdot)$ to denote the Jacobian, a non-linear map into a matrix target, and $df_{(\cdot)}(\cdot)$ to denote the differential, a linear map into a vector target, and he calls both a differential.


Another point why I find it confusing is that for me the Jacobian is a matrix of partial derivatives and the differential is an operator. For example, to differentiate the matrix function $f:\mathbb{R}^{2 \times 2} \rightarrow \mathbb{R}$:

$f(X) = tr AX$

I would use the differential operator:

$df(X; dX) = tr AdX$

And from the Jacobian identification table (Magnus19), I'll get:

$Df(X) = A'$

Note that the differential isn't a trivial linear map anymore.

It also leads to another point. The differential has a linear approximation meaning. Basically, it denotes the change in the function. If it's a scalar value function, the change would be scalar, and thus the differential (would map to a scalar). If the domain is matrices, then the Jacobian is a matrix (a non-linear map from matrices to matrices). I definitely would find it confusing if someone would treat them the same.


Let's do another example, $f:\mathbb{R}^{2 \times 2} \rightarrow \mathbb{R}^{2 \times 2}$:

$f(X) = AX$

Using the differential operator:

$df(X; dX) = AdX$

$vec\ df(X; dX) = (I_2 \otimes A) vec\ dX$

From the Jacobian identification table:

$Df(X) = I_2 \otimes A$

In this case, I'm not sure I'd consider the differential $df$ and Jacobian $Df$ almost the same thing (I'm not so good with tensors). This is the root of my issue. It's not always a simple matrix multiplication, and one needs to be mindful about the difference between the differential and Jacobian.

Not to mention the second order differential and the Hessian identification.


I corresponded with a couple of Caltech guys who settled it for me, and I can live with that. To paraphrase:

Math is a living language like any other, it evolves and changes. As long as we clearly define the terms in the context, there shouldn't be a problem--call it whatever you want.

$\endgroup$
2
$\begingroup$

If $M\subset \mathbb{R}^n$ is an open set and $f:M\to \mathbb{R}^k$ is differentiable, then for $p\in M$ we have the derivative $d_pf:\mathbb{R}^n\to\mathbb{R}^k$, a linear map. In your situation it is not necessary (but certainly possible) to think about tangent spaces. The matrix that describes the linear map $d_pf$ with respect to the standard bases of $\mathbb{R}^n$ and $\mathbb{R}^k$ I would denote by $f'(p)$ (you call it $J$). So it is just the matter of applying a linear map to a vector versus multiplying this vector by a matrix: $$d_pf(Y)=f'(p).Y$$ Almost the same thing...

$\endgroup$
4
  • $\begingroup$ That's fine, but I asked about the differential. Please see my added "EDIT2". $\endgroup$ – Zohar Levi Jul 24 '20 at 22:31
  • $\begingroup$ I think I'm starting to understand the confusion. For you, differential is synonymous with derivative. For me, differential is what you call variation (your $\delta$ operator in Chao10). $\endgroup$ – Zohar Levi Jul 25 '20 at 2:10
  • $\begingroup$ So your function $f$ depends on an additional variable $t$, so it really is a function of three variables? And your "differential" is the partial derivative with respect to $t$? For me, that would be the meaning of a "variation". And this is usually the way how can make sense of the $\delta$-notation. And yes, this notation is used Chao et al. where I am one of the authors, but please do not blame me. I hate that notation. $\endgroup$ – Ulrich Pinkall Jul 26 '20 at 7:33
  • $\begingroup$ I assumed the fancy math in appendix B in that paper was your doing. $\endgroup$ – Zohar Levi Jul 26 '20 at 22:42
2
$\begingroup$

I use the word "differential" (aside from the abuse the term gets in beginning calculus texts) only when referring to $1$-forms. So, of course, for a mapping $f\colon M\to\Bbb R$, the differential $1$-form at $p$ ($df(p)$) coincides with the derivative at $p$ as a linear map $Df(p)\colon T_pM\to\Bbb R$. (Some people write this $df(p)$, $Df_p$, $df_p$, and who knows what else.) Sometimes you will see that for vector-valued functions $f\colon M\to\Bbb R^k$, some of us will refer to the differential as a vector-valued $1$-form; this, too, coincides with the derivative as a linear map.

$\endgroup$
5
  • $\begingroup$ Let's consider scalar-valued functions. I'm fine with your differential 1-form definition that maps a vector from $M$ to a scalar: en.wikipedia.org/wiki/One-form#Differential. I'm not clear, though, why you call it derivative/Jacobian/gradient, where these entities map a vector to a vector. For example, do you agree with the definitions and answers ($df(2,2;v)=12$, $\nabla f(1,1)=(2,2)$) in math.stackexchange.com/questions/3071033/… $\endgroup$ – Zohar Levi Jul 25 '20 at 1:13
  • $\begingroup$ Gradient is out of play here. It’s the vector dual to the derivative/differential. That answer was sloppy because it confused a row vector (linear map) and column vector (e.g.. gradient). $\endgroup$ – Ted Shifrin Jul 25 '20 at 2:13
  • $\begingroup$ I agree (like Magnus19) that the gradient (column vector) is the transpose of the Jacobian (row vector). Therefore, up to transposition, they represent the derivative. The main point is, though, that these are vectors while the differential is a scalar (and I chose to use $\nabla$ to distinguish it from $d$). $\endgroup$ – Zohar Levi Jul 25 '20 at 2:23
  • $\begingroup$ No, the differential is not a scalar. It's a linear map. $\endgroup$ – Ted Shifrin Jul 25 '20 at 3:41
  • $\begingroup$ Fine; does it map to a scalar or a vector? What about the Jacobian/derivative? $\endgroup$ – Zohar Levi Jul 25 '20 at 7:52
1
$\begingroup$

No. The derivative of the map $f$ is the Jacobian, $J$ : $$ \frac{\mathrm{d}f}{\mathrm{d}x} = J \text{.} $$ Then the relation between the differentials is "algebra". (It's not. It's a lot of machinery for handling linear approximations. But it looks like algebra due to a judicious choice of notation.)

$\endgroup$
11
  • $\begingroup$ "No" what? And the Leibniz's notation "algebra" that you are referring to may hold for univariate, scalar functions, but not in general. Here, $df$ and $dx$ are vectors. $\endgroup$ – Zohar Levi Jul 21 '20 at 6:27
  • $\begingroup$ @ZoharLevi : There is only one question in the Question and it is the last sentence: "However, if f is considered a "mapping", then the differential of the mapping df is equal to the Jacobian J?" The answer to that question is "No." $\endgroup$ – Eric Towers Jul 21 '20 at 13:19
  • $\begingroup$ @ZoharLevi : The derivative of the map $f$ with respect to its input is the local linear approximation $J$. This is a map from displacement vectors (from $\vec{x}$) in the tangent space (at $\vec{x}$) of the domain space of $f$ to displacement vectors (from $f(\vec{x})$) in the tangent space (at $f(\vec{x})$) of the codomain of $f$. In your setting, it is the linear map $J$. $\endgroup$ – Eric Towers Jul 21 '20 at 13:25
  • $\begingroup$ @ZoharLevi : The "algebra" is not even true for univariate, scalar functions. The indecomposable object $\frac{\mathrm{d}f}{\mathrm{d}x}$ is not a ratio of differentials. As I wrote in the Answer, both equations are correct, but one does not pass between them by operations in some field, but instead by first defining derivatives and differentials, then showing that it is a fortuitous feature of our notation that something as simple as a (formal) field operation can syntactically transform the one equation into the other, (continued...) $\endgroup$ – Eric Towers Jul 21 '20 at 13:31
  • $\begingroup$ @ZoharLevi : but only by disregarding the type mismatch between the derivative $J$ and the differential $J \, \mathrm{d}x$. $\endgroup$ – Eric Towers Jul 21 '20 at 13:33
1
$\begingroup$

Short answer: if $f$ is a differentiable, then $Df(x)$ is the linear map and, if $f$ has continuous partial derivatives, then $Jf(x)$ is the matrix representation of the linear map $Df(x)$. That's all.

I will elaborate.

Let $f:\mathbb R^n \to \mathbb R^m$ be a differentiable function (we could restrict to open sets). By definition, this means that for every $x\in\mathbb R^n$ there is a linear map $T:\mathbb R^n \to \mathbb R^m$ such that

$$\lim_{y\to x}\frac{\|f(y)-f(x)-T\|}{\|y-x\|}=0.$$

One can prove that the linear map $T$ is uniquely specified by $f$ and $x$, so we can use the notation $Df(x):\equiv T$. This linear map is called the differential of $f$ at $x$ or the derivative of $f$ at $x$.

Let $f_1,\dots,f_m$ be the component functions of $f:\mathbb R^n\to\mathbb R^m$. This is, for every $x\in\mathbb R^n$ we have $f(x)=(f_1(x),\dots,f_m(x))$. Then for every point $x$, we define the as the matrix $Jf(x)\in M_{m\times n}\mathbb R$ whose $i$th row and $j$th column is the number $\partial_jf_i(x)$. One can also prove that if the partial derivatives $\partial_if_j$ are all continuous, then $Jf(x)$ is the matrix that represents the linear map $Df(x):\mathbb R^n\to\mathbb R^m$ with respect to the standard bases of $\mathbb R^n$ and $\mathbb R^m$.

With regards to notation, sometimes a lowercase $d$ is used instead of $D$. It is also convenient not to abuse the notation and distinguish clearly between $Df$, $Df(x)$, $J$, $Jf$ and $Jf(x)$. If we denote the set of linear maps $\mathbb R^n\to\mathbb R^m$ by $L(\mathbb R^n,\mathbb R^m)$, then $Df$ is the function $\mathbb R^n\to L(\mathbb R^n,\mathbb R^m)$ given by $x\mapsto Df(x)$. Similarly, $Jf$ is the function $\mathbb R^n\to M_{m\times n}\mathbb R$ given by $x\mapsto Jf(x)$. Personally I don't like to use $J$ by itself because it can give rise to confusion as to which object we are talking about.

$\endgroup$
1
  • $\begingroup$ Okay, but I asked about the differential. Please see my added "EDIT 2". $\endgroup$ – Zohar Levi Jul 24 '20 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.