2
$\begingroup$

These are the gradient descent formulas: $$\frac{\delta }{\delta m} = \frac{2}{n}\sum-x_i(y_i - mx_i + b)\\ \frac{\delta }{\delta m} = \frac{2}{n}\sum-(y_i - mx_i + b) $$ And my understanding is they come from first taking the positive gradient is the partial derivatives of the function $ (y - mx + b)^2$.

This leads to $$\frac{\delta J}{\delta m}( 2x(y - mx +b ))\\ \frac{\delta J}{\delta b}(2 \times (y - mx + b) \times 1)$$

Then to get the descent, we just add negatives to each partial derivative. So we are already descending.

But translating gradient descent into code, this is what I often see:

def linear_regression(X, y, m_current=0, b_current=0, epochs=1000, 
    learning_rate=0.0001):
    N = float(len(y))
    for i in range(epochs):
   y_current = (m_current * X) + b_current
   cost = sum([data**2 for data in (y-y_current)]) / N
   m_gradient = -(2/N) * sum(X * (y - y_current))
   b_gradient = -(2/N) * sum(y - y_current)
   m_current = m_current - (learning_rate * m_gradient)
   b_current = b_current - (learning_rate * b_gradient)
   return m_current, b_current, cost

My question is about the update to m_current and b_current in the final lines of the function. Why is m_current - learning rate * m_gradient and b_current - learning_rate * b_gradient?

Why not

  m_current = m_current + (learning_rate * m_gradient)
  b_current = b_current + (learning_rate * b_gradient)

Our gradient descents are already negative to point us towards descending along the cost curve, so why aren't we updating our m_current and b_current by just adding the respective gradients?

$\endgroup$
3
  • $\begingroup$ In the code, m_gradient and b_gradient are the two components of the gradient, which points in the direction of steepest ascent. $\endgroup$
    – littleO
    Commented Mar 4, 2018 at 1:40
  • $\begingroup$ So then where does negative sign come from for the m_gradient and b_gradient. $\endgroup$
    – user924088
    Commented Mar 4, 2018 at 1:57
  • $\begingroup$ Your expression for the gradient included minus signs also. I added an answer with your own minus signs highlighted. $\endgroup$
    – littleO
    Commented Mar 4, 2018 at 2:40

2 Answers 2

4
$\begingroup$

If we add the gradient to it, we are taking the step that is the steepest ascent, that is we are increasing our value locally, but here we are trying to minimize our objective function.

For example, suppose $y=2x$, the gradient is $2$, no matter where we are, adding $2$ to our current location increases the value of $y$.

$\endgroup$
2
  • $\begingroup$ yea - but aren't we already taking the negative when we go from (2x(y−mx+b)) to - (2x(y−mx+b)). In the code, this is four lines from the bottom. So why do we do this again in the second to last line? $\endgroup$
    – user924088
    Commented Mar 4, 2018 at 1:44
  • $\begingroup$ I agree with littleO that the gradient itself has a negative sign and m_gradient and b_gradient indeed are the corresponding gradient (rather than their negative). $\endgroup$ Commented Mar 4, 2018 at 3:23
1
$\begingroup$

m_gradient and b_gradient in the code are the two components of the gradient, which points in the direction of steepest ascent.

The objective function that we want to minimize is $$ J(m,b) = \frac{1}{N}\sum_{i=1}^N (y_i - (m x_i + b))^2. $$ Notice that I have included parentheses around $m x_i + b$. As you mentioned, the partial derivatives of $J$ are \begin{align} \require{color} \frac{\partial J(m,b)}{\partial m} = \frac{2}{N} \sum_{i=1}^N \colorbox{yellow}{-}x_i(y_i - (m x_i + b)), \\ \frac{\partial J(m,b)}{\partial b} = \frac{2}{N} \sum_{i=1}^N \colorbox{yellow}{-}(y_i - (m x_i + b)). \end{align} You included those highlighted minus signs yourself already. These expressions agree perfectly with m_gradient and b_gradient in the code.

$\endgroup$
1
  • $\begingroup$ got it, thanks! $\endgroup$
    – user924088
    Commented Mar 4, 2018 at 6:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .