Minor issues in documentation and traces for Nelder Mead

[eirikbrandsaas]

Hi,

While trying to understand what is going on in my own code I found some minor errors. I'll fix a few of them in a pull request, but other's I tried but wasn't able too.

• Some of the traces are not mentioned at https://julianlsolvers.github.io/Optim.jl/stable/#user/minimization/. (centroid_trace, simplex_trace, simplex_values_trace)
• function simplex_value_trace(r::MultivariateOptimizationResults) doesn't work, missing tr = trace(r)
• The output from simplex_trace and simplex_values_trace only give the values for the last iteration:

For the last one, here is a MWE:

using Optim
f(x) = (1.0 - x[1])^2 + 100.0 * (x[2] - x[1]^2)^2
x0 = [0.0, 0.0]
res = optimize(f, x0,iterations=3,show_trace=true,store_trace=true,extended_trace=true,trace_simplex=true );

Optim.centroid_trace(res) # Works
Optim.simplex_trace(res) # Doesn't work

julia> res = optimize(f, x0,iterations=3,show_trace=true,store_trace=true,extended_trace=true,trace_simplex=true );
Iter     Function value    √(Σ(yᵢ-ȳ)²)/n 
------   --------------    --------------
     0     9.506641e-01     4.576214e-02
 * simplex_values: [1.0, 0.9506640624999999, 1.0625]
 * time: 0.0009310245513916016
 * step_type: initial
 * centroid: [0.0125, 0.0]
 * simplex: [[0.0, 0.0], [0.025, 0.0], [0.0, 0.025]]
     1     9.506641e-01     2.023096e-02
 * simplex_values: [1.0, 0.9506640624999999, 0.9793678283691405]
 * time: 0.0014400482177734375
 * step_type: outside contraction
 * centroid: [0.0125, 0.0]
 * simplex: [[0.0, 0.0], [0.025, 0.0], [0.018750000000000003, -0.0125]]
     2     9.506641e-01     2.172172e-02
 * simplex_values: [0.9262175083160399, 0.9506640624999999, 0.9793678283691405]
 * time: 0.0019030570983886719
 * step_type: expansion
 * centroid: [0.021875000000000002, -0.00625]
 * simplex: [[0.065625, -0.018750000000000003], [0.025, 0.0], [0.018750000000000003, -0.0125]]
     3     9.262175e-01     5.243757e-02
 * simplex_values: [0.9262175083160399, 0.9506640624999999, 0.8292372137308122]
 * time: 0.0023620128631591797
 * step_type: expansion
 * centroid: [0.045312500000000006, -0.009375000000000001]
 * simplex: [[0.065625, -0.018750000000000003], [0.025, 0.0], [0.09843750000000001, -0.0031250000000000028]]

julia> Optim.centroid_trace(res)
4-element Array{Array{Float64,1},1}:
 [0.0125, 0.0]
 [0.0125, 0.0]
 [0.021875000000000002, -0.00625]
 [0.045312500000000006, -0.009375000000000001]

julia> Optim.simplex_trace(res)
4-element Array{Array{Array{Float64,1},1},1}:
 [[0.065625, -0.018750000000000003], [0.025, 0.0], [0.09843750000000001, -0.0031250000000000028]]
 [[0.065625, -0.018750000000000003], [0.025, 0.0], [0.09843750000000001, -0.0031250000000000028]]
 [[0.065625, -0.018750000000000003], [0.025, 0.0], [0.09843750000000001, -0.0031250000000000028]]
 [[0.065625, -0.018750000000000003], [0.025, 0.0], [0.09843750000000001, -0.0031250000000000028]]

[eirikbrandsaas]

@pkofod Pinging you to check on a) whether you agree on the issue and b) if so to take a look at the pull request. Sorry for nagging!

[SimonEnsemble]

it looks like this hasn't been merged? I am also frustratingly finding that simplex_trace just repeats the simplex from the last iteration, inconsistent with the printed trace that shows the simplex is changing.

[SimonEnsemble]

simplex_value_trace returns an error: UndefVarError: tr not defined.

[SimonEnsemble]

another issue might be: show_trace=true seems to be printing step_type: reflection for EVERY step. seems impossible since my simplex is shrinking. a bug?

[pkofod]

If you have these issues, some variable must not be updating or it's always referencing the same. Let's get this fixed :)

[eirikbrandsaas]

@pkofod I had an old pull request that I accidentally deleted that solved the first two problems (#942). I think those were very easy to fix but the third was too hard :)

Hope this gets fixed, I run into it every now and then when I want to really dive into the numerical solutions.

[pkofod]

I should be able to fix these this week :)

[pkofod]

another issue might be: show_trace=true seems to be printing step_type: reflection for EVERY step. seems impossible since my simplex is shrinking. a bug?

This seems wrong?

julia> optimize(rosenbrock, zeros(2),Optim.Options(show_trace=true,extended_trace=true)).trace
Iter     Function value    √(Σ(yᵢ-ȳ)²)/n 
------   --------------    --------------
     0     9.506641e-01     4.576214e-02
 * time: 0.00015997886657714844
 * step_type: initial
 * centroid: [0.0125, 0.0]
     1     9.506641e-01     2.023096e-02
 * time: 0.0005447864532470703
 * step_type: outside contraction
 * centroid: [0.0125, 0.0]
     2     9.506641e-01     2.172172e-02
 * time: 0.0007708072662353516
 * step_type: expansion
 * centroid: [0.021875000000000002, -0.00625]
     3     9.262175e-01     5.243757e-02
 * time: 0.0009908676147460938
 * step_type: expansion
 * centroid: [0.045312500000000006, -0.009375000000000001]
     4     8.292372e-01     4.259749e-02
 * time: 0.0012059211730957031
 * step_type: reflection
 * centroid: [0.08203125, -0.010937500000000003]
     5     8.292372e-01     4.265507e-02
 * time: 0.0014410018920898438
 * step_type: reflection
 * centroid: [0.11875000000000001, -0.012500000000000004]
     6     8.138907e-01     3.109209e-02
 * time: 0.0016717910766601562
 * step_type: reflection
 * centroid: [0.13515625, -0.004687500000000004]
     7     7.569606e-01     3.215435e-02
 * time: 0.0018999576568603516
 * step_type: reflection
 * centroid: [0.1515625, 0.003124999999999996]
     8     7.382898e-01     2.418419e-02
 * time: 0.002157926559448242
 * step_type: reflection
 * centroid: [0.16796875, 0.010937499999999996]
     9     6.989376e-01     2.426367e-02
 * time: 0.002610921859741211
 * step_type: reflection
 * centroid: [0.18437499999999998, 0.018749999999999996]
    10     6.800415e-01     2.124416e-02
 * time: 0.002895832061767578
 * step_type: reflection
 * centroid: [0.20078124999999997, 0.026562499999999996]
    11     6.475000e-01     1.809652e-02
 * time: 0.0031828880310058594
 * step_type: reflection
 * centroid: [0.21718749999999998, 0.034374999999999996]
    12     6.377042e-01     2.151280e-02
 * time: 0.003487825393676758
 * step_type: reflection
 * centroid: [0.23359374999999996, 0.042187499999999996]
    13     5.977620e-01     1.648361e-02
 * time: 0.003780841827392578
 * step_type: reflection
 * centroid: [0.24999999999999994, 0.049999999999999996]
    14     5.977620e-01     2.780850e-02
 * time: 0.004036903381347656
 * step_type: reflection
 * centroid: [0.26640624999999996, 0.057812499999999996]
    15     5.476196e-01     2.429423e-02
 * time: 0.004307985305786133
 * step_type: inside contraction
 * centroid: [0.24609374999999994, 0.0671875]
    16     5.449374e-01     1.754792e-02
 * time: 0.0046079158782958984
[...]