# Understanding the mathematics behind backpropagation

I have just finished watching a lecture from Patrick Winston's AI course. In an attempt to understand the mathematics behind back-propagation, I have formulated a simplistic neural network as follows:

## Task

Given an input $x \in \{0, 1\}$, train a simple neural network to mimic the identity function, that is $f(x) = x$.

## Definitions

$x$ = the input

$T$ = the target function; in this case $T(x) = x$

$y$ = the desired output; $y = T(x)$

$A$ = the activation function; I'll use $A(x) = \frac{1}{1 + e^{-x}}$

$\hat{T}$ = the feed-forward function

$\hat{y}$ = the output of the network; $\hat{y} = \hat{T}(x)$

$E$ = the error function; I'll use $E(\hat{y}, y) = \frac{1}{2}(\hat{y} - y)^2$

$w$ = a weight $\in [0, 1]$

$\alpha$ = the learning rate

Note that $\hat{T}(x) = A(wx)$ in this very simplistic example.

## Formulating back-propagation via gradient descent

In a particular iteration $i$ of this network, let $x_i \in \{0, 1\}$ and $\hat{y}_i = \hat{T}(x_i)$. Now, the weight $w_i$ of this network needs to be adjusted such that $E(\hat{y}_i, y_i) > E(\hat{y}_{i+1}, y_{i+1})$. So, $\frac{\partial{E}}{\partial{w_i}}$ will give the rate of change function of $E$ with respect to $w_i$. Computing this partial: \begin{align} \tag{1}\frac{\partial{E}}{\partial{w_i}} &= \frac{\partial{E}}{\partial{\hat{y}_i}} \frac{\partial{\hat{y}_i}}{\partial{w}_i} + \frac{\partial{E}}{\partial{y_i}} \frac{\partial{y_i}}{\partial{w}_i}\\ \tag{2}&= (\hat{y}_i - y_i) \cdot \frac{\partial{\hat{y}_i}}{\partial{w}_i} - (\hat{y}_i - y_i) \cdot 0\\ \tag{3}&= (\hat{y}_i - y_i) \cdot \frac{\partial{\hat{y}_i}}{\partial{w}_i}\\ \tag{4}&= (\hat{y}_i - y_i) \cdot \frac{\partial{\hat{T}}}{\partial{w}_i}\\ \tag{5}&= (\hat{y}_i - y_i) \cdot \frac{\partial{A(w_ix_i)}}{\partial{w}_i}\\ \tag{6}&= (\hat{y}_i - y_i) \cdot x_i \cdot (1 - A(w_ix_i)) \cdot A(w_ix_i) \end{align}

## The confusion

How does finding $\frac{\partial{E}}{\partial{w_i}}$ help in the gradient descent algorithm in this case? I only understand that the gradient of a function at a point returns a vector pointing in the direction of the greatest incline. How can the weight $w_i$ be updated in such a manner?

• The gradient descent algorithm minimizes a function $f(x)$ by repeatedly taking small steps in the direction of steepest descent:$x^{k+1}=x^k - t \nabla f(x^k)$, where the "stepsize" $t$ is a small positive number. Commented Jul 2, 2017 at 23:03
• Also $(1)$ follows from the "total derivative" $g(t) = f(u_1(t),\ldots,u_n(t)) \implies g'(t) = \sum_{i=1}^n u_i'(t) \frac{\partial f}{\partial u_i}(u_1(t),\ldots,u_n(t))$ Commented Jul 2, 2017 at 23:46

## 1 Answer

Once you have found the gradient $\frac{\partial{E}}{\partial{w_i}}$, you change the weight as follows:

$w_i (new) = w_i (old) -\mu \cdot \frac{\partial{E}}{\partial{w_i}}$

with $\mu$ being a small positive number (the learning rate)

So: if $\frac{\partial{E}}{\partial{w_i}}$ is positive, then increasing the weight will increase the error, so you want to move the weight in the opposite direction, i.e. subtract a little bit from the weight (and if $\frac{\partial{E}}{\partial{w_i}}$ is negative, then this formula will add a little bit to the weight).

We of course do not know how much to change the weight, so we only change it a little bit (the learning rate $\mu$ is usually set to be fairly small). So, we 'nudge' all weights in the 'right' direction (the direction that should decrease the error), and then we repeat the process.

Finally, we change the weight proportional to $\frac{\partial{E}}{\partial{w_i}}$, for that will follow the steepest descent in the 'error landscape'. Another way of thinking about that: we could change all weights with the same (little) amount into the right direction (up or down, again depending on the sign of the derivative), but because this is such a non-linear system we are dealing with, we can anticipate that changing weights may be good for certain input-output patterns, but bad for others. So, we want to change as 'little' as possible in terms of 'total change'. So: if we find a large $\frac{\partial{E}}{\partial{w_i}}$, then we know that changing that weight will have a large effect in decreasing that error (at least locally), so we'd rather change that weight than changing a weight that would have far less effect on the error.

• Is $\frac{\partial{E}}{\partial{w_i}}$ producing a direction of the steepest incline or is it just giving a magnitude of how reliant $E$ is on $w_i$? If so, how do we know that subtracting will get us in the direction of a local minimum? I think I am not understanding this fully. Commented Jul 2, 2017 at 23:11
• @ljeabmreosn yes if $f : \mathbb{R}^n \to \mathbb{R}$ then $-\nabla f(x) = -(\frac{\partial f}{x_1}(x),\ldots,\frac{\partial f}{x_n}(x))$ is the direction of the steepest descent at the point $x$, this is because $\lim_{h \to 0}\frac{f(x+hv)-f(x)}{h} =\sum_{i=1}^n v_i \frac{\partial f}{x_i}(x)$ and under the constraint $\|v\| = 1$ this is minimized when $v = -\nabla f(x)$ Commented Jul 2, 2017 at 23:42
• Okay, so if I understand correctly, if $\frac{\partial{E}}{\partial{w_i}} > 0$, then we decrease $w_{i+1}$ by some amount and vice versa if $\frac{\partial{E}}{\partial{w_i}} < 0$. So does that mean that what we increase/decrease by doesn't matter as long it is the same sign as $\frac{\partial{E}}{\partial{w_i}}$? Commented Jul 2, 2017 at 23:54
• @ljeabmreosn The direction of change will always be 'away' from the error, yes. The amount with which we change depends on the learning rate. Again, we don't know how much we should change the weight to get to a minimum, so we just play it safe and typically keep the aount fairly small, also because the direction may change after that little step. Imagine a river finding its way down a complicated landscape .. It snakes its way down bit by bit ... We try to do the same with this method. Indeed, by keeping the amount f change small, we are more likely to find the minimum. Commented Jul 3, 2017 at 1:09
• @ljeabmreosn You're welcome! :) Commented Jul 3, 2017 at 1:48