How to handle routes with logic to different Clusters?

[donaldgray]

What is the best way to conditionally choose a different Cluster to proxy request to?

For example, ideally I'd like to have an application with 2 Clusters (foo and bar) and a route with "clusterId": "foo" and I want to, depending on some logic applied to an incoming request, use cluster bar. Is this possible?

I'm currently using the Direct Forwarding but that means I lose out on load balancing to Clusters. I can apply any logic I need and change path/destination but without load balancing etc. Is it possible to get access to something that handles this, similar to how IHttpForwarder is used? With this method it looks like I'd need to send the request to another load-balancer for bar and let that do the load balancing.

I had a look at ITransformFactory but that looks like it is transforming requests/responses to/from a specified Cluster, rather than changing the target cluster.

Is this possible?

[Tratcher]

Can you give an example of the selection criteria you'd use to pick the cluster? Today routing must be used to select a cluster, and that's only so flexible.

IHttpForwarder is the most flexible for dynamic decisions. Our load balancing algorithms are pretty simple, you should be able to replicate them:

reverse-proxy/src/ReverseProxy/LoadBalancing/PowerOfTwoChoicesLoadBalancingPolicy.cs

Lines 29 to 35 in 219a2fe

	// Pick two, and then return the least busy. This avoids the effort of searching the whole list, but
	// still avoids overloading a single destination.
	var destinationCount = availableDestinations.Count;
	var random = _randomFactory.CreateRandomInstance();
	var first = availableDestinations[random.Next(destinationCount)];
	var second = availableDestinations[random.Next(destinationCount)];
	return (first.ConcurrentRequestCount <= second.ConcurrentRequestCount) ? first : second;

[donaldgray]

Can you give an example of the selection criteria you'd use to pick the cluster?

The example application serves images (on /image/{**req}). The consumer of the application can request various sizes, rotations, crops (according to IIIF Image spec). These requests are proxied to an image-server, which parses the request and generates the desired image.
We also have pregenerated thumbnails for popular sizes. Thumbnails can be requested directly on (on /thumb/{**req}) but I'd like to redirect /image/{**req} if we know we can satisfy the image with a known thumbnail as there's a lot less work going on to satisfy that request.

So, an example config could be:

"Routes": {
  "thumbnail": {
    "ClusterId": "thumb",
    "Match": {
      "Path": "/thumb/{**catch-all}",
    }
  },
  "Clusters": {
    "img": {
      "Destinations": {
        "img/one": { "Address": "http://localhost:5001", },
        "img/two": { "Address": "http://localhost:5002", }
      }
    },
    "thumb": {
      "Destinations": {
        "thumb/one": { "Address": "http://localhost:5011", },
        "thumb/two": { "Address": "http://localhost:5012", }
      }
    }
  }
}

Then something like the Direct Forwarding for implementing logic:

var clusters = endpoints.ServiceProvider.GetService<ICanGiveYouClusters>();

endpoints.Map("/image/{**req}", async httpContext =>
{
    var parsed = ImageRequest.parse(httpContext.Request.Path);
    if (IsRequestForKnownThumbSize(parsed)
    {
        var targetCluster = clusters.GetCluster("thumb");
        await targetCluster.Client.SendAsync(httpContext, requestOptions, transformer);
    }
    else
    {  
        var targetCluster = clusters.GetCluster("img");
        await targetCluster.Client.SendAsync(httpContext, requestOptions, transformer);
    }

Thanks for the link to LB algorithms - I'll have a look at those and replicate.

[Tratcher]

Interesting. One way to achieve that would be a url rewrite before routing.

app.Use((context, next) =>
{
  if (context.Request.Path.StartsWithSegments("/image", out remainder) && IsRequestForKnownThumbSize(remainder))
  {
    context.Request.Path = "/thumb" + remainder;
  }
  return _next();
});

[samsp-msft]

@Tratcher - Should we make ProxyConfigManager public? I could potentially then write code along the lines of:

        public Task ImageRedirect(HttpContext context, Func<Task> next)
        {
            var parsed = ImageRequest.parse(httpContext.Request.Path);
            if (IsRequestForKnownThumbSize(parsed))
            {
                var cfg = context.RequestServices?.GetService(typeof(ProxyConfigManager)) as ProxyConfigManager;
                var proxyFeature = context.Features.Get<IReverseProxyFeature>();
                proxyFeature.AvailableDestinations = cfg._clusters["img"].Destinations.Values.ToList();
            }
            return next();
        }

This step could then be put at the front of the pipeline and would change the list of destinations.

[Tratcher]

Should we make ProxyConfigManager public?

Making the whole config manager public just to get to the clusters seems like exposing too much surface area. We do have IClusterChangeListener to get clusters, but you have to track the resulting list yourself. There's room for improvement there.

You'd also need to update more than the available destinations list, the rest of the request state is still pointing at the config for the other route and cluster. If we want to make this kind of retargeting first class then I think we'd want an API you could call to swap everything together.

[samsp-msft]

I think switching out the clusters is not going to be uncommon (think A/B) scenarios so its something we need to have a better story for. Please excuse any typo’s this message was fat fingered on my phone.

…

________________________________ From: Chris Ross ***@***.***> Sent: Thursday, July 22, 2021 11:24:35 AM To: microsoft/reverse-proxy ***@***.***> Cc: Sam Spencer ***@***.***>; Comment ***@***.***> Subject: Re: [microsoft/reverse-proxy] How to handle routes with logic to different Clusters? (#1159) Should we make ProxyConfigManager public? Making the whole config manager public just to get to the clusters seems like exposing too much surface area. We do have IClusterChangeListener to get destinations, but you have to track the resulting list yourself. There's room for improvement there. You'd also need to update more than the available destinations list, the rest of the request state is still pointing at the config for the other route and cluster. If we want to make this kind of retargeting first class then I think we'd want an API you could call to swap everything together. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Freverse-proxy%2Fissues%2F1159%23issuecomment-885136648&data=04%7C01%7Csam.spencer%40microsoft.com%7C52c6bfe6019948b0eebc08d94d3df5b1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625750781560700%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4lWUuPfgETjOkBbo4hjtrJKbAcNGNmrgxZMQj5kFV%2FM%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FANC7AWRQYYSKFW4B4NTYLBLTZBO6HANCNFSM5AYG4IZA&data=04%7C01%7Csam.spencer%40microsoft.com%7C52c6bfe6019948b0eebc08d94d3df5b1%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637625750781570659%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zrPH44iNQBls3ovri7x2Ls2v6EY9c5ODfrJfSXobZIs%3D&reserved=0>.

[Tratcher]

True, this dynamic cluster selection is pretty similar to A/B requirements (#126). We might resolve this as a duplicate.

[donaldgray]

One way to achieve that would be a url rewrite before routing.

I tried this out this morning but I have a routing requirement where I want to route to X rather than Y but the path remains the same. For that reason I think I'll use the "Direct Forwarding" approach and mimic any loadbalancing logic.

The dynamic cluster A/B requirements look promising, I'll subscribe to that issue and keep an eye on how it progresses. Also, the final comment on that is about getting customers to help validate the design - I'd be more than happy to help out if I can.

[karelz]

Triage: Interesting idea, we should find out how many more customers would use it. Moving to Backlog for now.

[samsp-msft]

Currently you either have to consume the whole proxy pipeline including routing, or you go to the forwarder. There is nothing in between which would use the proxies load balancing, affinity, health checks etc. If you want to put in logic that changes the list of destinations, you can only do that by messing with the AvailableDestinations from a cluster.

In doing some more thinking on this general problem, I think we need to provide a "hook" to be able to adjust where the request is routed to. We currently don't have a good way to insert a new step in there, and routing has typically already picked a route, so could be too late. But if you handle the routing yourself, we then could have a nice entry point to tell the proxy to handle the request.

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));
var app = builder.Build();

app.Map("{**catch-all}", requestDelegate);

app.Run();

Task requestDelegate(HttpContext context)
{
    var proxy = builder.Services[IReverseProxy];
    
    // Enumerates all the current routes known to the proxy. Is a read-only collection of RouteModel. Method to indicate results are transient.
    var routes = proxy.GetRoutes();

    foreach (var route in routes)
    {
        if (route.Config.RouteID == "Juniper")
        {
            // Uses the proxy to forward a request to the given route - or should this just be to a name?
            return proxy.Forward(context, route);
        }
    }


    // alternatively can pick a cluster - this is good for any form of A/B scenario
    var westCoast = proxy.getCluster("westCoast");
    var eastCoast = proxy.getCluster("eastCoast");
    var destCluster = (resolveIPLocation(context.Connection.RemoteIpAddress).Longitude <= -120) ? westCoast : eastCoast;

    return proxy.Forward(context, destCluster, /* transforms */ null );

}

The concepts included in this are:

• A proxy service object that provides access to config snapshots including routes and clusters
• A forward method that will take a route, or cluster and forward the current request to it

[Tratcher]

We added something for this called ReassignProxyRequest. See https://microsoft.github.io/reverse-proxy/articles/ab-testing.html