Benchmarks

[pkofod]

My main use-case for this is benchmarking Optim. This package currently allows the user to solve a CUTEst.jl problem. Do you, @timholy , also see the scope containing the examples we have in the problems folder https://github.com/JuliaOpt/Optim.jl/tree/9d82523820a7a2674ac142c3074b7fc41eeebf83/src/problems? Then we need to either have two sets of functions, or dispatch on something else than a string. We could create cutest and problem string literals (cutest"SENSORS", problem"Himmelblau", ...) or maybe something even smarter.

Also, if we want to log benchmarking of unconstrained and constrained optimization, should it happen here, or in OptimBenchmarks.jl (doesn't exist yet).

[timholy]

As it is this is basically a CUTEst/Optim wrapper now, but it doesn't have to stay that way. I definitely think that benchmarking is within scope (indeed I hope to do some of that too).

Your API proposal makes sense. I'd lean slightly towards just having functions that return an OptimizationProblem from a CUTEst name, and then a common framework for running OptimizationProblems. Obviously we'd have to expand the fields inside OptimizationProblem to include the constraints, and I think it would be good to include the minimum as well as the minimizer.

[pkofod]

Okay, our thinking seems to be quite aligned then. Quite a few of the cutest problems have NaNs (or the function herein returns that due to no solution being found), at least for the unconstrained problems.

Then we'd have something like a

get_cutest_problem(cute_string::AbstractString)::OptimizationProblem

and we could have the same for the "pure julia" problems we have in Optim right now

get_native_problem(native_string::AbstractString)::OptimizationProblem`

(don't get too upset about the names, I'm aiming at the functionality right now, but I tend to be very verbose in my coding style). The result of either would then be passed to a

solve_problem(nlp::OptimizationProblem)

So what is an optimization problem? I guess something like

immutable OptimizationProblem
    name
    objective
    constraints
    initial_x
    minimum
    minimizer
end

Here objective is something you can call value, gradient!, and so on on. Remember, quite a few canonical, annoying unconstrained test problems have several local minima, sometimes with the same objective value (so they're all global minima). Should we have explicit \mathcal{C}^N-information on the objective and constraints here?

[timholy]

Yes, the availability of derivatives seems worth encoding.

[pkofod]

So when the other PR is merged (I know I made it more difficult than necessary myself), I will try to add a logging feature. So unless you have anything lying around already, I'll just use what I had inhttps://github.com/JuliaNLSolvers/Optim.jl/pull/259 and I can get comments on that approach.