Support reduced-space formulation

[Robbybp]

The documentation says:

MathOptAI.jl implements the full-space formulation. It cannot model a reduced-space formulation. We choose the full-space formulation for simplicity, and because, in our experience, "simpler" models with more variables and constraints are faster to formulate, compute derivatives of, and solve.

I believe this is actually an open research question for nonlinear problems. I've seen full and reduced space formulations each perform better for certain applications. I propose that we have an option to implement reduced-space formulations.

Longer term, it might even be nice to have a fully "black-box" formulation, where we just return to the user an external function for evaluating the predictor (along with its derivatives, if requested). This way we could potentially evaluate the predictor using something like PyTorch or Jax, which could take advantage of existing infrastructure for AD or GPU acceleration.

[odow]

I'm open to this, and at various times I had a prototype. I ended up going with simplicity in this initial version.

The reduced-space is actually quite easy to implement, since we just return expressions:

MathOptAI.jl/src/predictors/Affine.jl

Lines 55 to 69 in 4a8f1f3

	function add_predictor(model::JuMP.Model, predictor::Affine, x::Vector)
	m = size(predictor.A, 1)
	y = JuMP.@variable(model, [1:m], base_name = "moai_Affine")
	bounds = _get_variable_bounds.(x)
	for i in 1:size(predictor.A, 1)
	y_lb, y_ub = predictor.b[i], predictor.b[i]
	for j in 1:size(predictor.A, 2)
	a_ij = predictor.A[i, j]
	lb, ub = bounds[j]
	y_ub += a_ij * ifelse(a_ij >= 0, ub, lb)
	y_lb += a_ij * ifelse(a_ij >= 0, lb, ub)
	end
	_set_bounds_if_finite(y[i], y_lb, y_ub)
	end
	JuMP.@constraint(model, predictor.A * x .+ predictor.b .== y)

I wasn't sure whether the syntax should be:

add_predictor(model, predictor, x; reduced_space::Bool)

or

add_predictor(model, predictor, x)
add_reduced_space_predictor(model, predictor, x)

Probably the kwarg option that propagates the kwarg to nested predictors, and there is a fallback to the full-space implementation if not supported (e.g., ReLU big M?).

[odow]

See #71. I'd appreciate your thoughts.

[Robbybp]

At first thought, I'd probably go with

add_predictor(model, predictor, x; formulation::FormulationEnum=FULL_SPACE)

Thanks for opening #71. I'll try to give it a review tomorrow.

[odow]

I've also opened #72. I think I might like that better.

[odow]

it might even be nice to have a fully "black-box" formulation, where we just return to the user an external function for evaluating the predictor (along with its derivatives, if requested)

I think this is just a user-defined operator built-in to JuMP? Or are you thinking that we would automate the derivative info, etc.

[odow]

Closed by #72

Feel free to open a new issue for the black-box thing if you wanted something other than a user-defined operator.

[Robbybp]

Yes, I was thinking we could automate derivative info. I don't have a concrete use case for now though, but will re-open if it comes up.