3
$\begingroup$

In the gradient descent algorithm say $f(x)$ (quadratic function) is the objective function. SO the algorithm is defined as

$$x_i = x_i - a\frac{\partial f(x)}{\partial x_i}$$

I Just dont quite understand the meaning of doing a subtraction. I'm intuitively able to follow that we are going in the direction of steepest descent but have some questions. The derivative of $f(x)$ is going to give us the equation of a line. So when we substitute the value of $x_i$ in $f'(x)$ , what we get is a $y$ coordinate: $y_i$. So I dont understand how we subtract a $y$ coordinate from an $x$ coordinate ?

$\endgroup$

3 Answers 3

5
$\begingroup$

The direction of $\nabla f$ is the direction of greatest increase of $f$. (This can be shown by writing out the directional derivative of $f$ using the chain rule, and comparing the result with a dot product of the direction vector with the gradient vector.) You want to go toward the direction of greatest decrease, so move along $-\nabla f$.

$\endgroup$
2
  • $\begingroup$ Hi yes , I was able to get the general Idea. So $\nabla f$ gives us the equation of a straight line. And when we substitute the value of $X_i$ in that straight line equation we get a yi coordinate . So are we subtracting this yi coordinate from $X_i$ which is an X coordinate? $\endgroup$
    – karthik A
    Commented Sep 7, 2012 at 5:34
  • $\begingroup$ Okay I figured it out. Thanks !! $\endgroup$
    – karthik A
    Commented Sep 7, 2012 at 5:41
1
$\begingroup$

regarding So I dont understand how we subtract a 𝑦 coordinate from an 𝑥 coordinate ?

∂𝑓(𝑥)/∂𝑥 is not really a coordinate.

Maybe this example with simple scalar might help.

enter image description here Image/Answer Source

As you can see from this example, when the derivative is positive, you need to subtract the fraction of that derivative if you want to minimize the cost function. So, in the maximization problem, you would add the alpha * the derivative (slope).

Ok, this is for the scalar example. What about the multi-dimensional?

The same logic applies to the multi-dimension. In the case of more than one dimension, the gradient of a function would just be a vector of all its partial derivatives. So basically nothing changed.

$\endgroup$
0
0
$\begingroup$

My understanding of this minus sign is about the assumption of SGD. The assumption is that the objective function $J$ is a convex function where has the optimal solution (global, local) at $\theta_{*}$ where the partial derivatives are $0$, so that's why the parameters are updated by moving to the reverse direction of function "changing faster", because SGD wants $J$ changes slower and slower and gradually hits the convex point.

$\endgroup$
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .