This is what's written:

enter image description here

So, $h(x)$ is basically a linear function to predict values of some training set. I understand everything that's happening up to the point where we take the partial derivative of $h(x)$ in summation form (the second equation above the black line), but how do we get the partial derivative of $h(x) - y$ to just $x_j$?

Also, why does the update in the equation under the black line change what's calculated before: $(h(x) - y)x_j$ to $(y - h(x))x_j$?

I'm sorry if I'm not being clear enough. Please let me know if that's the case.

Thank you!


The partial derivative of $h$ is with respect to $\theta_i$. So we just follow normal differentiation rules. In this case, $x_i$ and $y$ are not functions of $\theta_i$, they are fixed constant. Also, since $\theta_i$ is not a function of $\theta_j$, it is held constant as well. So $\frac{\partial}{\partial \theta_j} (\theta_j x_i-y) = x_i$. It's just like taking $\frac{d}{dx} (ax+b) = a$.

In short:

$$\frac{\partial \theta_i}{\partial \theta_j} = 0$$ $$\frac{\partial x_i}{\partial \theta_j} = 0$$ $$\frac{\partial y}{\partial \theta_j} = 0$$ $$\frac{\partial \theta}{\partial \theta_j} = 1$$

Combine as needed using standard differentiation rules.

Also, the update step reverses the order because in the first equation, the update step subtracts off the Jacobian. The second equation adds the negative of that, which means that the terms inside the parenthesis get reversed: $a-b = -(b-a)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.