Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This is what's written:

enter image description here

So, $h(x)$ is basically a linear function to predict values of some training set. I understand everything that's happening up to the point where we take the partial derivative of $h(x)$ in summation form (the second equation above the black line), but how do we get the partial derivative of $h(x) - y$ to just $x_j$?

Also, why does the update in the equation under the black line change what's calculated before: $(h(x) - y)x_j$ to $(y - h(x))x_j$?

I'm sorry if I'm not being clear enough. Please let me know if that's the case.

Thank you!

share|cite|improve this question
up vote 1 down vote accepted

The partial derivative of $h$ is with respect to $\theta_i$. So we just follow normal differentiation rules. In this case, $x_i$ and $y$ are not functions of $\theta_i$, they are fixed constant. Also, since $\theta_i$ is not a function of $\theta_j$, it is held constant as well. So $\frac{\partial}{\partial \theta_j} (\theta_j x_i-y) = x_i$. It's just like taking $\frac{d}{dx} (ax+b) = a$.

In short:

$$\frac{\partial \theta_i}{\partial \theta_j} = 0$$ $$\frac{\partial x_i}{\partial \theta_j} = 0$$ $$\frac{\partial y}{\partial \theta_j} = 0$$ $$\frac{\partial \theta}{\partial \theta_j} = 1$$

Combine as needed using standard differentiation rules.

Also, the update step reverses the order because in the first equation, the update step subtracts off the Jacobian. The second equation adds the negative of that, which means that the terms inside the parenthesis get reversed: $a-b = -(b-a)$

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.