Is it possible programmatically to iterate over all suitable optimizers?

[lesshaste]

Say I want test how well all global optimizers that don't take derivatives and do support box constraints perform after 50 function evaluations on my 3d function, is there any way programmatically to select just those optimizers?

[timmens]

Hey, thanks for reaching out!

You can get a list of all (available) algorithms that fit your criteria using:

import optimagic as om

list_of_algos = om.algos.Scalar.Global.GradientFree.Bounded.Available

If you want to see all supported optimizers that fit these criteria, and not just the ones you have installed in your environment, simply write .All instead of .Available.

You can then use them in your minimization using

for algo in list_of_algos:
    om.minimize(
        fun=fun,
        params=params,
        algorithm=algo,
        algo_options={"stopping_maxfun": 50, "stopping_maxiter": 50},
    )

Some things to consider:

1. Some algorithms do not support the stopping_maxfun option. In this case, try to limit the maximum number of evaluations using the stopping_maxiter argument.
2. Some algorithms do not support any of these two arguments (e.g., "scipy_brute"). You must think about what you want to do in this case (drop them or add another argument that governs their behavior to the algo_options dictionary).
3. For more information you can check out our How to select a local optimizer guide. Even though it is written for local optimizers, many ideas translate to the global case.

[lesshaste]

Thank you. I tried it out with:

import optimagic as om
import numpy as np

def three_minima_function(params):
    x, y = params
    return (
        -2 * np.exp(-((x - 1) ** 2 + (y - 1) ** 2) / 0.2)
        - 1.5 * np.exp(-((x + 1) ** 2 + (y + 1) ** 2) / 0.5)
        - 1 * np.exp(-((x - 1) ** 2 + (y + 1) ** 2) / 1)
    )


list_of_algos = om.algos.Scalar.Global.GradientFree.Bounded.Available

bounds = om.Bounds(lower=np.array([-5, -5]), upper=np.array([5, 5]))

for algo in list_of_algos:
    res=om.minimize(
        fun=three_minima_function,
        params=np.arange(2),
        algorithm=algo,
        bounds=bounds,
        algo_options={"stopping_maxfun": 5, "stopping_maxiter": 5},
    )
    print(algo, three_minima_function(res.x), res.nfev)

In this case list_of_algos equals:

[om.algos.bayes_opt,
 om.algos.nlopt_crs2_lm,
 om.algos.nlopt_direct,
 om.algos.nlopt_esch,
 om.algos.nlopt_isres,
 om.algos.scipy_brute,
 om.algos.scipy_differential_evolution,
 om.algos.scipy_direct]

om.algos.bayes_opt, om.algos.scipy_brute, om.algos.scipy_differential_evolution and om.algos.scipy_direct all seem to ignore the limit on the number of function evaluations. That is I get:

om.algos.bayes_opt -0.9995906766777223 31
om.algos.scipy_brute -1.3228533299559655 800
om.algos.scipy_differential_evolution -2.018451751539757 201
om.algos.scipy_direct -0.5614175224534719 11

You already warned me about scipy_brute, but is there anything that can be done about the other ones?

[timmens]

Hey again! Now, in general I would recommend studying the parameter descriptions on our Optimizers page. We are currently in the processing of updating it, so you can expect an even better Options-description in the future. For your current problem:

• scipy_brute: You can control the number of grid points with the n_grid_points option. To better understand its behavior, I would refer you to the SciPy Docs.
• scipy_direct: If you study the Scipy Docs you see that the maximum number of function evaluations is only approximate. If you increase stopping_maxfun to a larger number, say >20, then the actual number is much closer to the requested number.
• scipy_differential_evolution: In the options description we explain how the maximum number of iterations influces the number of function evaluations. You can set the population_size_multiplier to a small number to reduce the function evaluations. By default a polishing step is performed, which runs the LBFGSB algorithm afterwards. Setting this to False also reduces them.
• bayes_opt: Good catch! We actually did not harmonize the option name here (I will look into this). For now you can use the option name n_iter. Note also that the number of function evaluations is also influenced by the init_points which governs the number of initial points that are sampled from the function.

Given that you identified the "problems" in your list of algorithms, and that some of them have different option names, I would recommend constructing the list of algorithms manually:

list_of_algos = [
    # Can be governed using stopping_maxfun and stopping_maxiter
    om.algos.nlopt_crs2_lm,
    om.algos.nlopt_esch,
    om.algos.nlopt_isres,
    om.algos.scipy_direct,
    # Need extra treatment
    om.algos.scipy_brute(n_grid_points=2),
    om.algos.scipy_differential_evolution(
        stopping_maxiter=1,
        population_size_multiplier=0,
        polish=False,
    ),
    om.algos.bayes_opt(n_iter=5, init_points=4)
]

And increasing the number of function evaluations and iterations to something larger than 10.

[lesshaste]

Thank you!