I am new to machine learning and AI in general and had a quick question regarding the error function surface regarding a simple neural net: 2 input neural net

After reading the following wiki: https://en.wikipedia.org/wiki/Backpropagation

I understand that the error function for a specific input and output of this neural net could be described as:

$$E = (t-y)^2$$ where $t$ is the expected value and $y$ is the value of the output node in the neural net.

However, $y$ is a function of $w_1$, and $w_2$:

$$y=x_1w_1 + x_2w_2$$

This I believe must mean that the final error function should be:

$$E = (t - x_1w_1 - x_2w_2)^2$$

If I graph this function for specific values of $t, x_1$ and $x_2$, I am getting a surface that is a parabolic cylinder instead of a parabolic bowl. Is there anything that I might be looking at wrong? Thanks so much.

  • $\begingroup$ Please, use MathJax (i.e. LaTeX commands) for mathematical notations. $\endgroup$ – Taroccoesbrocco Jul 11 '18 at 8:01
  • $\begingroup$ The loss function is chosen to be quadratic in the output variable $y$. That is a convention that is also used in other machine learning situations, notably in linear regression. That does not mean that it is quadratic in all the parameters as you correctly observed. It is certainly not a quadratic function of the weights leading into the hidden layer. $\endgroup$ – Hans Engler Jul 11 '18 at 14:33
  • $\begingroup$ This makes sense, however, is it not wrong to say that there is a discrepancy between the error function defined over $w_1$ and $w_2$ and the 3D plot shown in the wiki? $\endgroup$ – user2214288 Jul 11 '18 at 16:41
  • $\begingroup$ @user2214288 Yeah... if you look here, which shows how they generate the graph, they are plotting $z=x^2+y^2$, not $(x+y)^2$, I guess the description is off (?) $\endgroup$ – user3658307 Jul 14 '18 at 0:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.