# Analytic gradient, solving for non-convex neural network loss functions

Please, note that I am talking in theory here, rather than ''what someone would do in a real, practical situation''.

Given a multi-layer Perceptron with at least 1 hidden layer, and sigmoid (or other non-linearities), and a loss function, let it be a quadratic loss function such as $$L= ||\phi(X,W)-Y_{target} ||_{2}^2$$, where $$\phi(X,W)$$ is the output of the net, W are the weights, X the data matrix, $$Y_{target}$$ the correct labels. I think it does not make sense to solve for $$\nabla_W L = 0$$ to find a minimum ? I think it does not because, first of all, the function $$L$$, even though being ''the square of something'', is non-convex (possibly very complex with many ''hills'' and ''valleys'') and there can be many different values of $$W$$ giving a value of 0 for the gradient. Moreover, these points (W) where the gradient $$\nabla_W L = 0$$ could also be maxima rather than minima! Can someone with good theoretical background answer this/confirm/complete this please?

Yes, solving $$\nabla L(W) = 0$$ could result in a local minimum, or a local maximum, or a saddle point.
• Thanks for your answer! Also, if we tried to solve $\nabla_W L$, it is not clear what kind of results would it give, I mean "which" solution (local min, max, saddle) would it "choose" so-to-speak? (i tried in Matlab with a 1 hidden layer NN, never solved, had to ctrl-c ) I am puzzeld since we have N equations in N vars which sould give 1 result, right? Or would we sort of get "all solutions" at once (but in what form)? (N being the number of weights) Commented Jun 2, 2019 at 12:33
• Yes, if you used a black box method to find a solution to $\nabla L(W) = 0$, that method could return a local minimum or a local maximum or a saddle point. It is true that we have $N$ equations with $N$ unknowns, but the equations are nonlinear, so there is no guarantee that there will be a unique solution (and no reason to expect a unique solution). I think a black box solver would usually return one single solution, even though many solutions exist. Commented Jun 2, 2019 at 12:40
• I'm not surprised that a generic solver would fail to solve $\nabla L(W) = 0$ in the case of a neural network, because it's a challenging problem and we must exploit the special structure of the problem. Commented Jun 2, 2019 at 12:54
• Can we add that solving $\nabla_W L =0$ therefore only makes sense if L is convex? Commented Jun 2, 2019 at 13:03