# Gradient descent/ nonlinear optimization intuition needed

all. I'm taking an introductory AI class, and we're using the gradient descent algorithm to find the optimized/ lowest cost of a set of thetas (variable coefficients) to best fit a regression line. In the attached image (provided by course, please ignore markings from lecture) we are shown that to optimize all of the theta values, each is updated during each iteration of the gradient descent algorithm. Each should converge on its optimal value.

While I can implement this in octave, what I don't understand is, with multiple thetas being optimized, if one (or more) theta has reached its optimal value, but the algorithm continues to optimize the others, how does the already optimized theta not get altered/ moved away from its optimized value? In the slide, take theta1... if it is at its optimal value, and theta2 is still being optimized, in a given iteration, at theta 1, there is still a vector of error values (h(x) - y) of length m. Multiply this by each row's x value for that theta, and there will be a value, that is added to the current value of theta. So.. when a theta is optimized, but the algo continues, what makes a given theta value 'stay put?' Hope that makes sense. Thanks in advance.

Brief notation notes: alpha = scalar value to adjust degree of update, m = examples in data set, h theta(x) = the collective variable values * the current thetas, y = target variable in each row.

-
The value does not stay the same: if the other values are not at the optimum, the algorithm does not know that this one parameter already has the desired value. If the algorithm were moving in a straight line, the value would have to stay the same, but it looks more like a broken line. To better understand, you can try to plot what happens in the following situation: minimize $x^2 + xy + y^2$ and start at $(0,1)$. –  Vincent Zoonekynd Jun 9 '13 at 16:32
Thanks for the help! –  StatsViaCsh Jun 12 '13 at 13:09

I'm assuming that you're minimizing $J(\theta)$. The answer to your question consists in writing down what you mean by "one theta is already at its optimal value". In terms of optimization, it means that $$\frac{\partial}{\partial \theta_j} J(\theta) = 0.$$ When this equation is true for all $\theta$s, you know you're at (first-order) optimality.

But if the equation holds, then if you look at what you've framed in blue in your screenshot, you see that that $\theta_j$ isn't updated any more and stays put. Fortunately!

-