Proxy can divert routes to support A/B testing, service rollout etc

[Tratcher]

A/B testing is a common scenario for proxies, letting services incrementally upgrade/experiment using live traffic.

How does a service decide which requests to route to which test group?

• Assuming all requests are stateless you could decide this randomly on a per-request basis.
• For a stateful service you'd want to do a random assignment and then use affinization (Proxy supports Session Affinity to route subsequent requests to the same host #45).
• A service may also decide to statically affinitize based on known groups like tenants (e.g. move small customers before big ones).

Initial theory:

• This is a separate feature from load balancing. Within a load balancing set we assume all servers are equally able to handle a request aside from considerations of health and load. You'd select a test group and then load balance within that group.
• This should take advantage of the routing layer. For something like tenant based affinity that information may already be part of the route definition.
• Alternatively we could put it into the route config (ProxyRoute), allowing it to specify multiple backends and an assignment strategy.
• This should be compatible with retry mechanisms to allow failing over to the other test group.

[Tratcher]

@samsp-msft in your surveys have customers explained how they handle this today? And/or how they want to handle it?

[karelz]

Triage: Design should be done in 1.0 - customers will ask for it.

[Tratcher]

This could also come into play for rolling upgrades: #190 (comment)

1. Remove some nodes from pool1
2. Shut down those nodes
3. Upgrade and restart them
4. Add those nodes to pool2
5. Gradually transition a percentage of traffic from pool1 to pool2 and check for errors. (A/B testing)

Repeat until all nodes have been upgraded and moved.

[Tratcher]

Note: Backed.PartitioningOptions seems to be related to this scenario, but it's not actually implemented anywhere. Implement or remove this option section. The same for quota and circuit breaker options.

[Tratcher]

Notes from a partner discussion:

Routing - One Catch-All
 - Partitioning function to select cluster
Clusters, each with one destination
	- Next
	- Current
        - Previous

[Tratcher]

@johnazariah to your question from: #405 (comment)

You could still use the headers to decide where requests get assigned, but it would be applied at a stage after routing and would be much more customizable. Affinity is a definite question, the best option is to have a stable selection algorithm like the headers.

[johnazariah]

Ok I like what I'm hearing... 👍

[Tratcher]

Related: partitioning and shuffle sharding, great talk from AWS: https://www.youtube.com/watch?v=swQbA4zub20

[Tratcher]

E-Core3 is using IHttpProxy directly and does not need anything from us here. There is still general community interest so I'll do some prototyping and see what works.

[karelz]

Triage: Moving to post-1.0 -- we need engaged customers who can validate the design / E2E.

[Tratcher]

Here's a partial proposal for 1.1 that should let people build their own A/B features without us having to be too opinionated on their design:

Routes would need a default cluster to get past this check:

reverse-proxy/src/ReverseProxy/Model/ProxyPipelineInitializerMiddleware.cs

Lines 34 to 41 in 5b20508

	var cluster = route.Cluster;
	// TODO: Validate on load https://github.com/microsoft/reverse-proxy/issues/797
	if (cluster == null)
	{
	Log.NoClusterFound(_logger, route.Config.RouteId);
	context.Response.StatusCode = StatusCodes.Status503ServiceUnavailable;
	return Task.CompletedTask;
	}

In middleware re-assign the cluster:

reverse-proxy/src/ReverseProxy/Model/ProxyPipelineInitializerMiddleware.cs

Lines 43 to 50 in 5b20508

	var destinationsState = cluster.DestinationsState;
	context.Features.Set<IReverseProxyFeature>(new ReverseProxyFeature
	{
	Route = route,
	Cluster = cluster.Model,
	AllDestinations = destinationsState.AllDestinations,
	AvailableDestinations = destinationsState.AvailableDestinations
	});

We'd need a new service to get access to the clusters and routes.

• Read-only lookup by id: TryGetCluster/Route(id, out var cluster/route)
• enumeration

[Tratcher]

Can everyone here please take a look at #1538. It doesn't fully implement A/B or related systems, but it does provide some of the raw components to unblock people building their own.

[adityamandaleeka]

For 2.0 let's consider how to address the most common subset of use cases with extensibility points.

[karelz]

Triage: Needs Design flash out as next step. Collect requirements, make proposals, discuss them.