2
$\begingroup$

I am looking for a way how to prove/show that one problem is easier to optimize(more robust to starting guesses) than another.

Suppose I have two non-convex functions from $L_1,L_2:\mathbb{R_+}^8\rightarrow\mathbb{R}$, and I am trying to numerically find their minimum on an unconstrained support $\mathbb{R}_+$.

Specifically I am dealing with two types of log-likelihood functions for the same data. However, where $L_2$ seems agnostic to starting points, $L_1$ converges to different local minima.

I have investigated contour subplots of $L_1$, i.e. keeping all parameters fixed except varying two of them. On the plot below black X are starting points, blue diamonds are points marked as local optima by the solver and red points are those points where optimizer converged to absurdly large values.

Contour plot of L_1 showcasing convergence to different local minima (blue diamonds), points of failed convergence(red squares)

I notice that $L_2$ does not exhibit this behaviour (even though structurally is very similar to $L_1$) and therefore is more robust to the choice of starting guesses. I would like to get an idea why this happens. I have noticed that when comparing the two functions in some subspace of $\mathbb{R}^8$, $L_2$ is more "curved" than the other, as shown in a plot bellow. enter image description here

Can this be the reason and if so is there a mathematical (even numerical) way to show that one is more curved at a particular point than the other (perhaps using Hessian)?

Thanks!

$\endgroup$
  • $\begingroup$ Your plot for $L_1$ seems to be levelling off, even bending over. Does this function have another minima by any chance? or do you consider $\delta < \varepsilon$ change as converged? $\endgroup$ – Chinny84 Feb 6 at 15:22
  • $\begingroup$ Exactly, the function is bending over implicating its non-convexity. $L_1$ is confirmed to have multiple local minima in the full $\mathbb{R}^8$ space -- different starting guesses result in different points of convergence. By convergence I mean that the step size is either sufficiently small or step in any direction does not improve the obj. function. Thanks. $\endgroup$ – Michael Mark Feb 6 at 16:58
  • $\begingroup$ Ok, I am assuming that $L_1$ bends over in the vicinity of the initial conditions. But I am not completely sure, so I will default to another poster. $\endgroup$ – Chinny84 Feb 6 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.