23
$\begingroup$

Say $f$ is a scalar valued function from $\mathbb{R}^n \to \mathbb{R}$. When I learnt about the gradient $\nabla f(\mathbf{x})$ I always thought of it as a column vector in the same space as $\mathbf{x}$. That way, the dot product $\nabla f \cdot \mathbf{v}$ gives the directional derivative in direction $\mathbf{v}$.

All the definitions I can find of the Jacobian of $\mathbf{y} = \psi(\mathbf{x})$ however define it as:

\begin{bmatrix} \frac{\partial y_1}{\partial x_1} & \frac{\partial y_1}{\partial x_2} & \cdots & \frac{\partial y_1}{\partial x_n}\\ \frac{\partial y_2}{\partial x_1} & \frac{\partial y_2}{\partial x_2} & \cdots & \frac{\partial y_2}{\partial x_n}\\ \vdots&\vdots \\ \frac{\partial y_m}{\partial x_1} & \frac{\partial y_m}{\partial x_2} & \cdots & \frac{\partial y_m}{\partial x_n}\\ \end{bmatrix}

But this would make $\nabla f$ a row vector, which then means the directional derivative is no longer $\nabla f \cdot \mathbf{v}$.

Which way is correct? What are the consequences if I accidently write the Jacobian the opposite way? I have found some similar questions here but none that answer my question directly. I'm still learning this stuff so please explain in simple terms :)

$\endgroup$

3 Answers 3

25
$\begingroup$

In general, the derivative of a function $f : \mathbb{R}^n \to \mathbb{R}^m$ at a point $p \in \mathbb{R}^n$, if it exists, is the unique linear transformation $Df(p) \in L(\mathbb{R}^n,\mathbb{R}^m)$ such that $$ \lim_{h \to 0} \frac{\|f(p+h)-f(p)-Df(p)h\|}{\|h\|} = 0; $$ the matrix of $Df(p)$ with respect to the standard orthonormal bases of $\mathbb{R}^n$ and $\mathbb{R}^m$, called the Jacobian matrix of $f$ at $p$, therefore lies in $M_{m \times n}(\mathbb{R})$.

Now, suppose that $m=1$, so that $f : \mathbb{R}^n \to \mathbb{R}$. Then if $f$ is differentiable at $p$, $Df(p) \in L(\mathbb{R}^n,\mathbb{R}) = (\mathbb{R}^n)^\ast$ is a functional, and hence the Jacobian matrix, as you point out, lies in $M_{1 \times n}(\mathbb{R})$, i.e., is a row vector. However, by the Riesz representation theorem, $\mathbb{R}^n \cong (\mathbb{R}^n)^\ast$ via the map that sends a vector $x \in \mathbb{R}^n$ to the functional $y \mapsto \left\langle y,x \right\rangle$. Hence, if $f$ is differentiable at $p$, then the gradient of $f$ at $p$ is the unique (column!) vector $\nabla f(p) \in \mathbb{R}^n$ such that $$ \forall h \in \mathbb{R}^n, \quad Df(p)h = \left\langle \nabla f(p),h\right\rangle; $$ in particular, if you unpack definitions, you'll find that the Jacobian matrix of $f$ at $p$ is precisely $\nabla f(p)^T$.

$\endgroup$
6
  • 2
    $\begingroup$ Right, so there is only one correct way to write the Jacobian (the way I've written above) and defining it as the transpose is wrong? $\endgroup$
    – Flash
    Mar 21, 2013 at 8:16
  • 1
    $\begingroup$ Indeed. The Jacobian matrix at a given point has to be written as you wrote it to actually be the matrix (with respect to the usual bases) of the linear transformation that is the derivative at that point. $\endgroup$ Mar 21, 2013 at 8:18
  • 1
    $\begingroup$ Great, makes sense thanks $\endgroup$
    – Flash
    Mar 21, 2013 at 8:50
  • 2
    $\begingroup$ The gradient is the column vector $\nabla f(p)$, so that the Jacobian matrix is the row vector $\nabla f(p)^T$. $\endgroup$ Jan 19, 2017 at 12:14
  • 1
    $\begingroup$ @BranimirĆaćić, by your last comment, do you mean the gradient and Jacobian are different? Where could I read more about this? I am really confused about row vector vs column vector in terms of Jacobian, esp. when chain rule is involved. $\endgroup$
    – zyxue
    Oct 23, 2017 at 15:41
2
$\begingroup$

Imagine writing $\nabla f \cdot \mathbf{v}$ out. You would take the transpose of $\nabla f$ and put it next to $\mathbf{v}$ in order to carry out the dot product. So, really, $\nabla f$ is, and should be thought of as, a row vector. In other words, you're quite correct. It is, in fact, well depending on how you look at it, a convention of most multi-variable calculus textbooks that they write gradients as column vectors. I suppose I should mention the fancy, differential-geometric way of saying this is that the gradient of a scalar function is a covariant vector, and despite the informality with which I said it, this is actually the understanding behind the formalism of the gradient and its higher generalizations to manifolds.

It might help you out some to look at the Riesz representation theorem, or the bra-ket notation in quantum mechanics.

$\endgroup$
3
  • 2
    $\begingroup$ Maybe I'm nitpicking but how can $\nabla f \cdot \mathbf{v}$ mean the same thing if $\nabla f$ is a row vs a column vector? How can you even take the dot product of a row vector with a column vector? $\endgroup$
    – Flash
    Mar 21, 2013 at 6:57
  • 1
    $\begingroup$ Well at that point it just becomes matrix multiplication. If you adjoin a row vector with a column vector (in that order!) and carry out the product according to the usual rules of matrix multiplication, you get the dot product. Thus you can think of the "dot" as something that turns column vectors, say $a$, into row vectors $a \cdot$. Once you consider $a$ as a row vector, there essentially ceases to be a dot next to it. $\endgroup$
    – user66354
    Mar 21, 2013 at 7:11
  • $\begingroup$ One only writes the dot, as in $a \cdot$, if one still means by $a$ a column vector. Try to look up and understand the connections between a vector space, its dual space, and an inner product on the original vector space, I believe it's where your confusion lays, essentially. $\endgroup$
    – user66354
    Mar 21, 2013 at 7:11
0
$\begingroup$

EDIT: Uh, something's wrong, maybe I should have $Dy_i$s in the matrix below, pardon.

Here's perhaps another couple ways to look at it. This maybe should be a comment to the OP, but I'm not sure how to comment.

$$Jac({\bf y}) =\begin{bmatrix} \nabla y_1\\ \vdots\\ \nabla y_m \end{bmatrix} =\begin{bmatrix} {\partial\over\partial x_1}{\bf y}\cdots{\partial\over\partial x_n}{\bf y} \end{bmatrix} $$

$\endgroup$
2
  • $\begingroup$ Andrew mentions the dual and inner product. A covector is very much associated with row vectors. df is a covector. It is in the dual space to the vectors. If you have an inner product, you may set up an isomorphism, and then, and olny then, shall we define the gradient, $\nabla f$ $\endgroup$ Mar 21, 2013 at 7:46
  • $\begingroup$ In other words, $\nabla f\cdot v=df(v)$ $\endgroup$ Mar 21, 2013 at 7:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .