There are so many different optimization algorithms out there, and lots of research going on. However, I have difficulties to find good comparison between them, and all articles / books / papers seem to evaluate them in different ways.

Isn't there some set of standard functions to run "standardized" benchmarks in order to test and compare all such algorithms? Is there some reference website showing how each algorithm performs?

(I'm especially interested in numerical derivative-free optimization techniques)

  • $\begingroup$ What kinds of optimization problems are you interested in? For certain problem classes there are sets of standard problems. $\endgroup$ Sep 19 '11 at 20:02
  • $\begingroup$ In my case, I guess it is: continuous, unconstrained, non-differentiable, noisy, non-convex ...in other words: just some noisy function $\endgroup$
    – dagnelies
    Sep 19 '11 at 21:44
  • $\begingroup$ I don't know if there is such a site (or set of standard functions), but here are two reasons why such a site might not exist: 1) A lot of the optimization problems that are solved in practice have a great deal of structure to them, and so they are often solved by specialized algorithms that exploit that structure. Such algorithms may work poorly or not at all on general problems. 2) An algorithm's performance is often implementation-dependent, and so the same algorithm may work much faster or slower depending on all kinds of things related to the implementation. $\endgroup$ Sep 19 '11 at 22:25
  • $\begingroup$ Still, if you don't get an answer here and you end up finding one yourself later, would you mind posting it back here? I would be interested to know about such a site or set of standard functions. $\endgroup$ Sep 19 '11 at 22:25
  • $\begingroup$ I presume you've gone through these? $\endgroup$ Sep 19 '11 at 22:48

The CUTEr collection (https://magi-trac-svn.mathappl.polymtl.ca/Trac/cuter/wiki) bundles over 1000 optimization problems including the Hock and Schittkowski collection, the Maros and Meszaros collection, and more. Problems are modeled in the (a bit outdated) SIF modeling language. The main reference is http://dl.acm.org/citation.cfm?id=962439

EDIT: CUTEst, the updated version of CUTEr, is available from https://ccpforge.cse.rl.ac.uk/gf/project/cutest/wiki. The main reference is https://doi.org/10.1007/s10589-014-9687-3.

The collection is also available in the AMPL modeling language (along with lots of other problems): http://www.orfe.princeton.edu/~rvdb/ampl/nlmodels (see also https://github.com/mpf/Optimization-Test-Problems).

The COPS collection has discretized control problems: http://www.mcs.anl.gov/~more/cops/

On the same page, you will read about performance profiles which are a standard tool to compare several algorithms on a given problem collection.

You may also enjoy Performance World: http://www.gamsworld.org/performance

Hans Mittelmann maintains benchmark results for all sorts of optimization problems and solvers: http://plato.asu.edu/bench.html

Jorge Mor\'e has a website on benchmarking derivative-free optimization codes: http://www.mcs.anl.gov/~more/dfo/


Due to optimisation problems varying considerably you might want to creating a library of test problems that somewhat represent the problems you are trying to solve, optimising these using multiple optimisation algorithms and comparing their performance. For that an increasingly accepted method to compare optimisation software is performance profiles, as stated by @Dominique. A method to generate these in Python can be found here. For MATLAB see the post of @Dominique.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.