Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In the gradient descent algorithm say $f(x)$ (quadratic function) is the objective function. SO the algorithm is defined as

$$x_i = x_i - a\frac{\partial f(x)}{\partial x_i}$$

I Just dont quite understand the meaning of doing a subtraction. I'm intuitively able to follow that we are going in the direction of steepest descent but have some questions. The derivative of $f(x)$ is going to give us the equation of a line. So when we substitute the value of $x_i$ in $f'(x)$ , what we get is a $y$ coordinate: $y_i$. So I dont understand how we subtract a $y$ coordinate from an $x$ coordinate ?

share|cite|improve this question
up vote 3 down vote accepted

The direction of $\nabla f$ is the direction of greatest increase of $f$. (This can be shown by writing out the directional derivative of $f$ using the chain rule, and comparing the result with a dot product of the direction vector with the gradient vector.) You want to go toward the direction of greatest decrease, so move along $-\nabla f$.

share|cite|improve this answer
Hi yes , I was able to get the general Idea. So $\nabla f$ gives us the equation of a straight line. And when we substitute the value of $X_i$ in that straight line equation we get a yi coordinate . So are we subtracting this yi coordinate from $X_i$ which is an X coordinate? – karthik A Sep 7 '12 at 5:34
Okay I figured it out. Thanks !! – karthik A Sep 7 '12 at 5:41

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.