Solver and JuMP returns not the same

[jd-lara]

During some simulation testing with PowerSimulations, this issue appears arbitrarily where the Xpress status doesn't match the the return of JuMP.primal_status(jump_model). The models run in direct mode is this useful to know.

julia> JuMP.solution_summary(ed.internal.container.JuMPmodel)
* Solver : Xpress

* Status
  Result count       : 0
  Termination status : INFEASIBLE
  Message from the solver:
  "6 Global search complete - integer solution found ( XPRS_MIP_OPTIMAL). - no interruption - the solve completed normally"

* Candidate solution (result jump-dev/JuMP.jl#1)
  Primal status      : INFEASIBLE_POINT
  Dual status        : NO_SOLUTION
  Objective bound    : 0.00000e+00

* Work counters
  Solve time (sec)   : 6.38302e-01

Another interesting thing is that if I call optimize!() again on the model the return is now correct

Julia version

Julia Version 1.10.0
Commit 3120989f39b (2023-12-25 18:01 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (x86_64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Max
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, westmere)
  Threads: 1 on 10 virtual cores
Environment:
  JULIA_EDITOR = code
  JULIA_NUM_THREADS =

(test) pkg> st
Status `~/.julia/dev/PowerSimulations/test/Project.toml`
  [4c88cf16] Aqua v0.8.4
  [336ed68f] CSV v0.10.12
  [9961bab8] Cbc v1.2.0
  [a93c6f00] DataFrames v1.6.1
  [864edb3b] DataStructures v0.18.16
  [60bf3e95] GLPK v1.1.3
⌅ [87dc4568] HiGHS v1.1.2
  [fc1677e0] HydroPowerSimulations v0.8.0
  [2cd47ed4] InfrastructureSystems v1.22.1
⌅ [b6b21f68] Ipopt v1.4.0
  [0f8b85d8] JSON3 v1.14.0
  [4076af6c] JuMP v1.19.0
  [f28f55f0] Memento v1.4.1
⌅ [c36e90e8] PowerModels v0.20.1
  [bed98974] PowerNetworkMatrices v0.10.0
  [e690365d] PowerSimulations v0.27.0 `..`
  [f00506e0] PowerSystemCaseBuilder v1.2.3 `~/.julia/dev/PowerSystemCaseBuilder`
  [bcd98974] PowerSystems v3.2.2
  [295af30f] Revise v3.5.13
  [c946c3f1] SCS v2.0.0
  [e2f1a126] StorageSystemsSimulations v0.9.0
  [98d24dd4] TestSetExtensions v3.0.0
⌅ [9e3dc215] TimeSeries v0.23.2
  [a759f4b9] TimerOutputs v0.5.23
  [ade2ca70] Dates
  [56ddb016] Logging
  [44cfe95a] Pkg v1.10.0
  [9a3f8284] Random
  [9e88b42a] Serialization
  [8dfed614] Test
  [cf7118a7] UUIDs
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated`

[jd-lara]

@odow looks like an Xpress problem/bug

FICO Xpress v8.12.3, Hyper, solve started 17:59:10, Feb 7, 2024
Heap usage: 56MB (peak 174MB, 150MB system)
Minimizing MILP  with these control settings:
OUTPUTLOG = 1
MPSNAMELENGTH = 64
CALLBACKFROMMASTERTHREAD = 1
Original problem has:
     61920 rows        62174 cols      1522616 elements        48 globals
Presolved problem has:
      8426 rows         9347 cols       133858 elements        48 globals
LP relaxation tightened
Presolve finished in 0 seconds
Heap usage: 73MB (peak 174MB, 150MB system)

Coefficient range                    original                 solved        
  Coefficients   [min,max] : [ 3.19e-18,  1.00e+06] / [ 2.01e-05,  1.30e+05]
  RHS and bounds [min,max] : [ 1.39e-17,  1.00e+06] / [ 1.22e-05,  2.57e+09]
  Objective      [min,max] : [ 8.33e-03,  1.00e+06] / [ 1.02e-06,  5.12e+08]
Autoscaling applied Curtis-Reid scaling

Symmetric problem: generators: 37, support set: 48
 Number of orbits: 165, largest orbit: 2
 Row orbits: 310, row support: 620
Will try to keep branch and bound tree memory usage below 23.6GB
Starting concurrent solve with dual, primal and barrier (8 threads)

                           Concurrent-Solve,   0s
            Dual                      Primal                     Barrier      
    objective   dual inf       objective   sum inf        objective   sum inf 
 D -2.802E+08  1.104E+10 |  P  1315262.4  1.717E+09 |  P  7456563.2   .0169626
----- interrupted ------ | ----- interrupted ------ | ------ infeasible ------
Concurrent statistics:
      Dual: 5708 simplex iterations, 0.38s
    Primal: 9704 simplex iterations, 0.38s
   Barrier: 33 barrier and 779 simplex iterations, 0.38s
            Barrier used 8 threads 8 cores, L1\L2 cache: 64K\4096K
            Barrier used SSE2 support, crossover used 8 threads
Problem is infeasible
 *** Search completed ***
Problem is integer infeasible
Numerical issues encountered:
   Dual failures    :     77 out of       504 (ratio: 0.1528)
   Singular bases   :    133 out of     11429 (ratio: 0.0116)
   Predicted att. level   : 0.3707
User solution (_) dropped.
Uncrunching matrix
  Solution time / primaldual integral :         0s/ 100.000000%
  Number of solutions found / nodes   :         0 /         0
┌ Error: Optimizer returned INFEASIBLE_POINT
└ @ PowerSimulations ~/.julia/dev/PowerSimulations/src/core/optimization_container.jl:680
General IIS
 
Problem is MIP feasible

[odow]

Transferred this to Xpress.jl

[jd-lara]

@odow I have the sense that this issue and #210 are related

[odow]

Let me take a look.

[odow]

Can you dump the MPS file of a failing example?

[odow]

I don't understand this at all. The log makes it look like the problem is actually infeasible:

Problem is infeasible
 *** Search completed ***
Problem is integer infeasible

If the problem is actually infeasible. Then the only weird thing is the raw status string?

Can you try with set_attribute(model, "MOI_POST_SOLVE", false)

[odow]

If solving twice fixes it, then that really sounds like an Xpress bug. But it could also be in Xpress.jl... Impossible without an easily reproducible example.

[jd-lara]

It seems related to the use of the setting "MAXMEMORYSOFT" => 600000 in combination with problem modification. Once I removed this, the problem hasn't happened again. I added some capabilities in PSI to get infeasible problems written to disk

[odow]

I talked to @jd-lara. Until proven otherwise, we're marking this as an upstream bug in Xpress. One difficulty is that he's using an old version of Xpress, FICO Xpress v8.12.3, so we can't check if it still exists in the latest version.

[joaquimg]

That is very old indeed.
Looks like some numerical issue.
Xpress has a svf file that is great for debugging. You can write one before and another one right before the problem is first solved and send to fico

[odow]

I don't know if there is anything left to do here for Xpress.jl.

[odow]

Okay, so I took another look at this.

I've made a bunch of changes to the solution status handling, but I don't think that's the problem.

I think it might be because of:

Coefficient range                    original                 solved        
  Coefficients   [min,max] : [ 3.19e-18,  1.00e+06] / [ 2.01e-05,  1.30e+05]
  RHS and bounds [min,max] : [ 1.39e-17,  1.00e+06] / [ 1.22e-05,  2.57e+09]
  Objective      [min,max] : [ 8.33e-03,  1.00e+06] / [ 1.02e-06,  5.12e+08]
Autoscaling applied Curtis-Reid scaling

You have some very small coefficients that are then scaled to 1e-5, so it seems plausible that during the scaled barrier solve it might detect infeasibility, but then on re-solve (perhaps with simplex, or it doesn't scale) it finds a feasible solution.

You could try setting "XPRS_SCALING" => 0.

Regardless, I don't think this is a bug in Xpress.jl.

[odow]

Closing for now as won't-fix. If anyone runs into something similar with a recent version of Xpress, please comment and we can re-open.