Updating Benchmark Sets

[AdityaPandeyCN]

Checklist

• Appropriate tests were added
• Any code changes were done in a way that does not break public API
• All documentation related to code changes were updated
• The new code follows the
  contributor guidelines, in particular the SciML Style Guide and
  COLPRAC.
• Any new documentation only uses public API

Additional context

This PR adds the recently implemented SciPy global optimization algorithms from OptimizationSciPy.jl (#927) to the black-box global optimizer benchmarks.

[ChrisRackauckas]

It's going to need to wait for the registration to finish JuliaRegistries/General#133534 JuliaRegistries/General#133535

Also could you add the new PyCMA one which just merged around the same time? SciML/Optimization.jl#933

[AdityaPandeyCN]

Sure

[ChrisRackauckas]

https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3473#0197b061-b0dd-4ffe-8fc5-2e2d200a57c8

@mxpoch it looks like PyCMA is giving a segfault and having some IO issues as part of the benchmark. Does PyCMA have to do IO in order to give the results? Seems like it could just all be memory?

[AdityaPandeyCN]

Hi @ChrisRackauckas Do you think we should run the Python-based optimizers sequentially to avoid the errors, this maybe happening because of multithreaded Python calls in Julia?

[mxpoch]

@ChrisRackauckas Yep the default behaviour of PyCMA is to store logs in an 'outcmaes' folder. If you pass verb_log=0 as an argument it shouldn't do any writes.

[AdityaPandeyCN]

I have made changes to run the SciPy optimizers sequentially, I have tried to run this on my local machine but this became overwhelming for my machine:-

1. In one of the iteration the sphere function test ran successfully and moved to the ellipsoidal but I interrupted as it ran for hours, the sphere test function was the one which corrupted so maybe running sequentially can fix this.
2. I scaled down the iteration and they ran correctly but I don't think this verifies anything? as the error happened at approx 60 billion iterations(https://buildkite.com/julialang/scimlbenchmarks-dot-jl/builds/3474/steps/canvas?sid=0197b0e1-97d8-43ef-866b-8264b670d97c)
   Any suggestion @ChrisRackauckas

[ChrisRackauckas]

as the error happened at approx 60 billion iterations

That seems like it would be over the limit? Is the maxiters not being passed on?

[AdityaPandeyCN]

Are we referring to the run_length here?, if so than yes seems like the SciPy optimizers are ignoring it and we have to set an explicit maxiters here.