1
$\begingroup$

In gradient descent algorithms (especially when talking about multivariate regression), when they are talking about the gradient at a given point, sometimes I find the notation: $\nabla f(x_n)$ and sometimes it is the notation: $\frac{\partial}{\partial x_n} f$

For example:

  • Wikipedia says: $x_{n+1} = x_n - \alpha \nabla f(x_n) $
  • Andrew Ng (on Coursera) says: $\theta_j = \theta_j - \alpha \frac{\partial}{\partial \theta_j} J (\theta)$

So I was wondering if the two are the same, or if there differences when we should use one notation over the other.

$\endgroup$
  • $\begingroup$ In a Hilbert space, there is a correspondence between linear functionals and points in the space. The gradient is the point corresponding to the linear functional $h \mapsto {\partial f(x) \over \partial x} h$. $\endgroup$ – copper.hat Feb 13 '17 at 3:32
  • $\begingroup$ I'm just starting this, so for someone who doesn't know (yet) what a Hilbet space is, can I consider that these two notations are the same? $\endgroup$ – Ryan B. Feb 13 '17 at 3:33
  • 2
    $\begingroup$ If you are working in $\mathbb{R}^n$, then think of ${\partial f(x) \over \partial x}$ as a row vector and $\nabla f(x) = {\partial f(x) \over \partial x}^T$ as a column vector. $\endgroup$ – copper.hat Feb 13 '17 at 3:35
  • 1
    $\begingroup$ Thank you very much, it's way clearer now (I've spent hours on this). $\endgroup$ – Ryan B. Feb 13 '17 at 3:42
  • 1
    $\begingroup$ They contain exactly the same information in $\mathbb{R}^n$. $\endgroup$ – copper.hat Feb 13 '17 at 3:52
1
$\begingroup$

$\nabla f(x_n)$ does NOT mean $\partial f/\partial x_n$. In the context of the wikipedia article on gradient descent, $x_n$ is a just a point in (say) $\mathbb{R}^3$. For example $x_n$ could be the point (1,0,2).

$\nabla f(x_n)$ is the vector $(\frac{\partial f}{\partial x} (x_n),\frac{\partial f}{\partial y} (x_n),\frac{\partial f}{\partial z} (x_n))$.

$\partial f/\partial x_n$ implies that you are labelling the coordinates $x_1, x_2,$ etc. Maybe $x_n$ happens to correspond to the coordinate $z$ in the 3d case, then this would be $\partial f/\partial z$ (which you would evaluate at some particular point).

Basically, I think the wikipedia article is using vector notation and Ng is using coordinate notation, which are essentially the same (I'm not sure especially since there's a mistake in your given example). But the real issue is what you stated in your first line.

$\endgroup$
  • $\begingroup$ Thank you very much, I was confusing between evaluating the gradient at some point, and getting the gradient parameters. I just did my equations and now Wikipedia's notation and Ng's match up. $\endgroup$ – Ryan B. Feb 13 '17 at 4:32
  • $\begingroup$ Follow up question: Say $x$ is a vector. then what is the difference between $\frac{\partial f}{\partial x}$ and $\nabla_x f$? Does this change anything? $\endgroup$ – Toke Faurby May 18 '17 at 0:20
  • 1
    $\begingroup$ @TokeFaurby In that context they mean exactly the same thing. $\endgroup$ – mathematician May 18 '17 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.