I'm studying Neural Networks for machine learning (by Geoffrey Hinton's course) and I have a question about learning rule for linear neuron (lecture 3).

Linear neuron's output is defined as: $y=\sum_{i=0}^n w_ix_i$

Where $w_i$ is weight connected to input #i and $x_i$ is input #i.

Learning procedure consists of changing weight vector in such way, so the actual output $y$ becomes more and more close to target output $t$. For learning weights we use Delta rule: $$\Delta w_i=-\epsilon\frac{dE}{dw_i} =\epsilon x_i (t-y)$$ Where $\epsilon$ is a learning rate.

I don't understand 2 things:

1) why do we change weights proportionally to their error derivatives?

2) why do we put minus before $\frac{dE}{dw_i}$ (error change)?

Logically, if error increased then $\frac{dE}{dw_i}>0$ and, since we want to decrease the error, minus sign makes our $\Delta w_i$ negative and, hence, decreases $w_i$. But if $\frac{dE}{dw_i}<0$ and our error decreases, then we are increasing the weight? Why?

Thank you VERY MUCH for your help!


tl;dr if you want to understand what you are doing beyond following "the delta rule", you should review some basic vector calc and look up "gradient-descent" which is what you are doing.

We want to find $w$'s that minimize $E$ (think of $E(w)$ as a function of $w = (w_1, ...,w_n)$ since the inputs $x$ are held fixed for a given training set).

Recall that the gradient $\nabla E = (\frac{dE}{dw_1}, ...,\frac{dE}{dw_n})$ is the direction in the $w$ vars in which $E$ increases the most (this is a basic fact from vector calc).

Since we want to decrease $E$ optimally, we go in the opposite direction $-\nabla E$. We move from previous $w = (w_1,...,w_n)$ to next position $w -\epsilon \nabla E$. If you rewrite this more explicitly, you see that your are updating each coordinate in the following way:

$$w_i \mapsto w_i - \epsilon \frac{d E}{dw_i}.$$

The epsilon is just a constant that changes how "fast" we follow the direction in which the gradient is decreasing fastest. If you go too fast, you might skip past the minimum, and your algorithm will take too long to converge. If you go too slow you may also take too long to converge as well.

Note that without conditions on the function $E$ (i.e. convexity), there is no guarantee that this converges to a global minimum. Part of the magic of deep learning is that in most cases one can engineer their way past this theoretical impasse (there are also heuristic arguments one can make about avoiding local minima when dimensions are large but there no theoretical convergence proofs to my knowledge).

  • $\begingroup$ Thank you for great explanation! $\endgroup$ – Super-kenguru Dec 27 '17 at 16:25

The error in here is the quadratic error:

$$ E_k = (t_k - y_k)^2 = (t_k - \sum_i w_ix_i^k)^2 $$ for the training examples indexed with $k$. Actually, you can also sum over $k$, and you can take the gradient instead of a single derivative.

Question 1: Changing weights proportionally to the negative derivative always works, with suitably small gain factor $\epsilon$, if the error function is convex in the unknowns $w_i$. This is the case here, as the error is quadratic.

Question 2: the negative sign will always work for convex functions. In particular, you can test this with the quadratic function. This is a particular example of so called gradient descent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.