# Trying to understand LMS algorithm

This is what's written:

So, $h(x)$ is basically a linear function to predict values of some training set. I understand everything that's happening up to the point where we take the partial derivative of $h(x)$ in summation form (the second equation above the black line), but how do we get the partial derivative of $h(x) - y$ to just $x_j$?

Also, why does the update in the equation under the black line change what's calculated before: $(h(x) - y)x_j$ to $(y - h(x))x_j$?

I'm sorry if I'm not being clear enough. Please let me know if that's the case.

Thank you!

-

## 1 Answer

The partial derivative of $h$ is with respect to $\theta_i$. So we just follow normal differentiation rules. In this case, $x_i$ and $y$ are not functions of $\theta_i$, they are fixed constant. Also, since $\theta_i$ is not a function of $\theta_j$, it is held constant as well. So $\frac{\partial}{\partial \theta_j} (\theta_j x_i-y) = x_i$. It's just like taking $\frac{d}{dx} (ax+b) = a$.

In short:

$$\frac{\partial \theta_i}{\partial \theta_j} = 0$$ $$\frac{\partial x_i}{\partial \theta_j} = 0$$ $$\frac{\partial y}{\partial \theta_j} = 0$$ $$\frac{\partial \theta}{\partial \theta_j} = 1$$

Combine as needed using standard differentiation rules.

Also, the update step reverses the order because in the first equation, the update step subtracts off the Jacobian. The second equation adds the negative of that, which means that the terms inside the parenthesis get reversed: $a-b = -(b-a)$

-