1
$\begingroup$

Please, note that I am talking in theory here, rather than ''what someone would do in a real, practical situation''.

Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other non-linearities), and a loss function, let it be a quadratic loss function such as $L= ||\phi(X,W)-Y_{target} ||_{2}^2$, where $\phi(X,W)$ is the output of the net, W are the weights, X the data matrix, $Y_{target}$ the correct labels. I think it does not make sense to solve for $\nabla_W L = 0$ to find a minimum ? I think it does not because, first of all, the function $L$, even though being ''the square of something'', is non-convex (possibly very complex with many ''hills'' and ''valleys'') and there can be many different values of $W$ giving a value of 0 for the gradient. Moreover, these points (W) where the gradient $\nabla_W L = 0$ could also be maxima rather than minima! Can someone with good theoretical background answer this/confirm/complete this please?

$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Yes, solving $\nabla L(W) = 0$ could result in a local minimum, or a local maximum, or a saddle point.

But we can hope that methods such as gradient descent or stochastic gradient descent will find a good local minimum.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer! Also, if we tried to solve $\nabla_W L$, it is not clear what kind of results would it give, I mean "which" solution (local min, max, saddle) would it "choose" so-to-speak? (i tried in Matlab with a 1 hidden layer NN, never solved, had to ctrl-c ) I am puzzeld since we have N equations in N vars which sould give 1 result, right? Or would we sort of get "all solutions" at once (but in what form)? (N being the number of weights) $\endgroup$
    – SheppLogan
    Jun 2, 2019 at 12:33
  • $\begingroup$ Yes, if you used a black box method to find a solution to $\nabla L(W) = 0$, that method could return a local minimum or a local maximum or a saddle point. It is true that we have $N$ equations with $N$ unknowns, but the equations are nonlinear, so there is no guarantee that there will be a unique solution (and no reason to expect a unique solution). I think a black box solver would usually return one single solution, even though many solutions exist. $\endgroup$
    – littleO
    Jun 2, 2019 at 12:40
  • $\begingroup$ I'm not surprised that a generic solver would fail to solve $\nabla L(W) = 0$ in the case of a neural network, because it's a challenging problem and we must exploit the special structure of the problem. $\endgroup$
    – littleO
    Jun 2, 2019 at 12:54
  • $\begingroup$ oh ok thanks. Very interesting answer. $\endgroup$
    – SheppLogan
    Jun 2, 2019 at 13:02
  • $\begingroup$ Can we add that solving $\nabla_W L =0$ therefore only makes sense if L is convex? $\endgroup$
    – SheppLogan
    Jun 2, 2019 at 13:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .