# Are the “weights” inside a neural network actually “terms” for a polynomial?

This just hit me today. I am not too experienced with math or neural networks, but I am trying to find out about them in my own way so I can some day understand them well.

So I was thinking about how neural networks are connected to more familiar things that I know. This is just speculation, but I would like to know if I am on the right track at all.

Currently I think that neural networks are in fact "function adapters" (if that is the correct term in english) so that when they are learning, they are trying to adapt some invisible function so that the inputs match the wanted outputs.

If I wanted to do something like this by hand, I would of course adjust the terms of some function that I am adapting.

Like if I had a simple polynomial:

5x + 1


I would adjust the x until the function outputs what I want with the given input.

I think the x in this example might actually represent a weight in a neural network. It would make a lot of sense if this was the case!

And then there is the "back propagation", which I have not studied that much at all, but I think it has to do with correcting the other weights when adjusting one, because if the weights are the unknowns of a polynomial and I adjust some unknown in a polynomial - all the previous calculations would be off to account for this name input, because the old inputs use the same network / polynomial as the new input that the network / polynomial was just adjusted for. So this "back propagation" takes this into account and tries to minimize the error for the old inputs?

Am I on the right track here?

Simply put:

Are weights = unknowns in a polynomial

Is "back propagation" = Making sure that the polynomial gives same outputs for the old inputs, after the polynomial has been adjusted to work with a new input

Thanks!

• I guess you'd rather want to change the constants (instead of $x$) so that your polynomial is "close", in some sense, to the points that are the training data. Backpropagation (afaik) is a fancy term for cleverly applying the chain rule. – Peter May 13 '16 at 15:43

I think of neural networks as a construction kit for functions. The basic building block - called a "neuron" - is usually visualized like this:

It gets a variable number of inputx $x_0, x_1, \dots, x_n$, they get multiplied with weights $w_0, w_1, \dots, w_n$, summed and a function $\phi$ is applied to it. The weights is what you want to "fine tune" to make it actually work. When you have more of those neurons, you visualize it like this:

In this example, it is only one output and 5 inputs, but it could be any number. The number of inputs and outputs is usually defined by your problem, the intermediate is to allow it to fit more exact to what you need (which comes with some other implications).

Now you have some structure of the function set, you need to find weights which work. This is where backpropagation (which is only a clever implementation of gradient descent) comes into play. The idea is the following: You took functions ($\varphi$) which were differentiable and combined them in a way which makes sure the complete function is differentiable. Then you apply an error function (e.g. the euclidean distance of the output to the desired output, Cross-Entropy) which is also differentiable. Meaning you have a completely differentiable function. Now you see the weights as variables and the data as given parameters of a HUGE function. You can differentiate (calculate the gradient) and go from your random weights "a step" in the direction where the error gets lower. This adjusts your weights. Then you repeat this steepest descent step and hopefully end up some time with a good function.

For two weights, this awesome image by Alec Radford visualizes how different algorithms based on gradient descent find a minimum (Source with even more of those):

So think of back propagation as a shortsighted hiker trying to find the lowest point on the error surface: He only sees what is directly in front of him. As he makes progress, he adjusts the direction in which he goes.

More specific for your question: Yes, the weights of a neural network are similar to a function like $f(x_1) = w_1 x_1 + w_0$, where you adjust $w_0, w_1$ to make the output as similar to some given pairs of input / output as possible. But the function is obviously not a polynomial. It is much more complicated.

• Thank you. This was very interesting and helpful! I will mark my question as answered. – Piwwoli May 17 '16 at 11:38

Have a look at the artificial neuron article.

The general model to connect the inputs $x_j$ to the outputs $y_k$ weighted by the $w_{kj}$ factors is $$y_k = \varphi(w_{kj} x_j)$$ where $\varphi$ is some transfer function.

You would need a polynomial transfer function to end up with a polynomial in the $x_i$ for $y_k$.