# Gradient descent - why subtract gradient to update $m$ and $b$

These are the gradient descent formulas: $$\frac{\delta }{\delta m} = \frac{2}{n}\sum-x_i(y_i - mx_i + b)\\ \frac{\delta }{\delta m} = \frac{2}{n}\sum-(y_i - mx_i + b)$$ And my understanding is they come from first taking the positive gradient is the partial derivatives of the function $(y - mx + b)^2$.

This leads to $$\frac{\delta J}{\delta m}( 2x(y - mx +b ))\\ \frac{\delta J}{\delta b}(2 \times (y - mx + b) \times 1)$$

Then to get the descent, we just add negatives to each partial derivative. So we are already descending.

But translating gradient descent into code, this is what I often see:

def linear_regression(X, y, m_current=0, b_current=0, epochs=1000,
learning_rate=0.0001):
N = float(len(y))
for i in range(epochs):
y_current = (m_current * X) + b_current
cost = sum([data**2 for data in (y-y_current)]) / N
m_gradient = -(2/N) * sum(X * (y - y_current))
b_gradient = -(2/N) * sum(y - y_current)
m_current = m_current - (learning_rate * m_gradient)
b_current = b_current - (learning_rate * b_gradient)
return m_current, b_current, cost


My question is about the update to m_current and b_current in the final lines of the function. Why is m_current - learning rate * m_gradient and b_current - learning_rate * b_gradient?

Why not

  m_current = m_current + (learning_rate * m_gradient)
b_current = b_current + (learning_rate * b_gradient)


Our gradient descents are already negative to point us towards descending along the cost curve, so why aren't we updating our m_current and b_current by just adding the respective gradients?

• In the code, m_gradient and b_gradient are the two components of the gradient, which points in the direction of steepest ascent. Commented Mar 4, 2018 at 1:40
• So then where does negative sign come from for the m_gradient and b_gradient. Commented Mar 4, 2018 at 1:57

For example, suppose $y=2x$, the gradient is $2$, no matter where we are, adding $2$ to our current location increases the value of $y$.
m_gradient and b_gradient in the code are the two components of the gradient, which points in the direction of steepest ascent.
The objective function that we want to minimize is $$J(m,b) = \frac{1}{N}\sum_{i=1}^N (y_i - (m x_i + b))^2.$$ Notice that I have included parentheses around $m x_i + b$. As you mentioned, the partial derivatives of $J$ are \begin{align} \require{color} \frac{\partial J(m,b)}{\partial m} = \frac{2}{N} \sum_{i=1}^N \colorbox{yellow}{-}x_i(y_i - (m x_i + b)), \\ \frac{\partial J(m,b)}{\partial b} = \frac{2}{N} \sum_{i=1}^N \colorbox{yellow}{-}(y_i - (m x_i + b)). \end{align} You included those highlighted minus signs yourself already. These expressions agree perfectly with m_gradient and b_gradient in the code.