Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

This is the rule:
\begin{align*} \theta_j:=\theta_j+\alpha\sum_{i=1}^{m}(y^{(i)}-h_\theta(x^{(i)}))x_j^{(i)} \space (\forall j) \end{align*} I can't get the correct thetas, they quickly get to infinity or some very large numbers. My question is, for $\theta_0$, $x_0$ is always 1 right? And so, for $\theta_0$ the $x_j$ is not applicable.

Thank you (Merry Christmas)

share|cite|improve this question
More information please. – Alex Becker Dec 26 '11 at 10:59
Yes, I agree with Alex ; your question seems to be described as if you asked it to a teacher in the context of a course, i.e. where he understands all the notations you just used. We're not familiar with those, a little context would be appreciated. – Patrick Da Silva Dec 26 '11 at 11:14
I think this question better suited to signal processing. – Gigili Dec 26 '11 at 13:05
@Gigili thanks for the suggestion – Andrew Dec 26 '11 at 13:21
up vote 3 down vote accepted

This looks like multivariate gradient descent.
In this case, indeed, for convenience we choose $x_0=1$, more explicitly $x_0^{(i)}=1 \space \forall i$.
By making this choice the hypothesis function of our $x_1...x_n$ features: \begin{align*} h_\theta(x)=\theta_0+\theta_1x_1+\theta_2x_2+...\theta_nx_n \end{align*} can be conveniently written as: \begin{align*} h_\theta(x)=\theta_0x_0+\theta_1x_1+\theta_2x_2+...\theta_nx_n=\theta^Tx \end{align*} When applying gradient descent to fit our $\theta$ parameters in the step that "descends downhill": \begin{align*} \theta_j:=\theta_j+\alpha\sum_{i=1}^{m}(y^{(i)}-h_\theta(x^{(i)}))x_j^{(i)} \space (\forall j) \end{align*} for $j=0$, given $x_0=1$, we'll be having: \begin{align*} \theta_0:=\theta_0+\alpha\sum_{i=1}^{m}(y^{(i)}-h_\theta(x^{(i)})) \end{align*} As such, one may say that " for $\theta_0$ the $x_j$ is not applicable".

The reason you "quickly get to infinity or some very large numbers" is that your $\alpha$ parameter is too large and in this case gradient descent does not converge. You must find the sufficiently small $\alpha$ parameter for which the gradient descent converges, and theory says that it exists. Notice, that if $\alpha$ is too small, the algorithm may be too slow, so you want to find an $\alpha$ just small enough so that the gradient descent converges, not smaller.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.