I'm starting to study machine learning using Andrew Ng's class notes. I understand conceptually how linear regression works, but am having trouble with this equation:

$$ \theta_j := \theta_j + \alpha\left(y^{(i)} - h_\theta (x^{(i)})\right) x_j^{(i)} $$

This is of course the update rule used to obtain values of theta using gradient descent. The notes say that one of the properties of this equation is that the magnitude of the update is proportional to the error term (the part between the big parentheses).

But seems like the update would be biased by larger values of $x_j$ too, since we are multiplying the error term by the value of $x_j$. To give a simple example, if we have a dataset with 5 observations, which have $x_j$ values of 1 through 5, won't the observation with $x_j$=5 have 5 times as much 'impact' on the change in $\theta_j$ compared to the one with $x_j$ = 1? I'm guessing that I'm misunderstanding something about the algorithm or what the variable $x_j$ represents, but if someone could explain where I'm going wrong I'd greatly appreciate it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.