Need for a column showing a reference solution if it exists, or distance from a reference solution. Ideas?

[DoctorDro]

Dear all,

I have been running the OptimizationProblems for the last 2 weeks, constantly improving my NLP solver over unconstrained problems. In some cases however, even LBFGS goes to a wrong solution, my solver as well. It is quite hard to isolate visually these cases when looking at the final pretty_stats. I would like to initiate a brainstorming of ideas on what is the best way to introduce a column where we show the difference from a reference solution (the closest to zero the better).

These problems the unconstrained ones, have been there and have been used for ages. Somewhere there should be a database of the local solutions Newton or any other gradient-based solver converges from the included starting point .meta.x0. Don't you think that this should be included in the problem case? Another option would be to run Newton or the best solver first that converges for most problems and use the solution of Newton as reference. But since OptimizationProblems.meta have so much info about every problem, it makes sense to include a reference solution for people to compare their findings against.

If we can find this info, then the information printed by the Benchmarks should add next to the achieved objective the reference optimal objective and its dual infeasibility.

What do you think guys?

[tmigot]

I think this is more an issue for OptimizationProblems.jl .

I like the idea, but this is not a trivial thing to do.
We try to document as much as possible the functions for know solutions when it is possible.

If we were reporting one thing, I guess it is the best minimal objective value. In the meta there is an entry :best_known_lower_bound and :best_known_upper_bound with this target. Unfortunately, this has been added at a later stage and most of the problem indicates -Inf for the lower bound. So, I would say this is mainly work in progress.

Two things to keep in mind:
First, I don't know if this is known for all the problems here.
Then, most of the solvers you mention are local optimizers and they are not expected to find the global minimum. It is not even something they try to do. Moreover, if you run two local solvers with the same initial guess, it is very possible that they give you two different solutions and that would ok.

[DoctorDro]

Yes you are right this is an issue for OptimizationProblems.jl.

I think it is much easier than it sounds. I saw the :best_known_upper_bound, and I assumed it is really the maximum. But when I tried to maximize the problems, all of them go to +Inf. So I have no idea what this :best_known_upper_bound stands for.

It might not be known for all the problems, but newton with linesearch and inertia correction for unconstrained solves most of them. Some of the rest might be solved by "trunk" or another method in JSOSolvers.

I have seen local solutions from the starting point, but not all methods go to the same local. Newton usually does not. If a local solver converged to something different than Newton, then this is a clear message for the developer of the local solver to improve his method by one way or the other. You never know if this local solution is really a solution or an inflection point some gradient - based algorithm got stack into. Even in that case this is a good test.

[dpo]

even LBFGS goes to a wrong solution

What do you mean by that? Problems may have many solutions and many stationary points. Could you show the output of LBFGS?

Including a "reference" solution is not obvious, again because nonconvex problems may have many solutions and solvers typically only look for a stationary point.

The best known upper bound is a value $M$ that represents the smallest known objective value (found by a solver or documented in the literature). Thus, there exist values of $x$ such that $f(x) \leq M$.