Allow more Hessian set types than just Set{Tuple{Int,Int}}

[gdalle]

At the moment, using TracerSparsityDetector{G,H} with H something other than Set{Tuple{Int,Int}} will fail. But one might like to try:

1. G = MySet{Int} and H = MySet{Tuple{Int,Int}} (definitely)
2. G = MySet{Int} and H = MyOtherSet2Tuple{Int,Int}} (maybe)

To remedy this, here are the steps to take:

• Allow MySet to accept things beyond Number (easy)
• Implement union! for MySet (easy except for RecursiveSet)
• Make sure that the result of the Cartesian product x(::MySet1{Int}, ::MySet1{Int}) can be union-ed with MySet{Tuple{Int,Int}}. This can be solved in two non-mutually-exclusive ways:
  • Make x(::MySet{Int}, ::MySet{Int}) return a MySet{Tuple{Int,Int}}. That way, the existing homogeneous union(::MySet{T}, ::MySet{T}) will work
  • Implement heterogeneous unions union(::MySet{T}, ::AbstractSet{T}). I don't like this option because it will presumably be very slow, in a way that is not directly visible to the user.

[adrhill]

I've spent a lot of time thinking about this and think that the cleanest solution is to implement several methods

×(::Type{H}, grad1::G, grad2::G)

for HessianTracer{G,H} / TracerSparsityDetector{G,H}.

It solves options 1 and 2 and avoids some of the pitfalls of heterogeneous Hessian-set unions, since H is specified prior to computing a Hessian-set union.

[gdalle]

okay, but I'll start with the homogeneous methods

[adrhill]

And for maximum performance, we might even end up needing

×!(hessian_buffer::H, grad1::G, grad2::G)

in the long term to avoid allocations.