Support list of loadBalancerPolicy for HTTPProxy

[wildermesser]

Currently, I can use only one policy from:

• RoundRobin
• WeightedLeastRequest
• Random
• Cookie
  But envoy supports a list of parameters for hash_policy. And the cookie is only one of them. With cookie loadBalancerPolicy, I have this envoy's config:

                    "route": {      
                      "cluster": "cluster_name",
                      "hash_policy": [
                        {     
                          "cookie": {                                         
                            "name": "X-Contour-Session-Affinity",
                            "ttl": "0s",
                            "path": "/"
                          }
                        }
                      ]                                                       
                    }

When a request does not contain this cookie, it will be routed to a random backend server. But I want to set additional rules like:

                    "route": {      
                      "cluster": "cluster_name",
                      "hash_policy": [
                        {     
                          "cookie": {                                         
                            "name": "X-Contour-Session-Affinity",
                            "ttl": "0s",
                            "path": "/"
                          }
                        },
                        "terminal": "true",
                        {     
                          "header": {                                         
                            "header_name": "User-Agent",
                          }
                         },
                          {
                          "connection_properties": {                                         
                            "source_ip": "true",
                          }
                        }
                      ]                                                       
                    }

It would be cool when Contour can support this rich load balancing functionality.
I understand that my case may be very specific. What is the simplest way to change runtime envoy configuration which will not changes when changing pod's list?
Should I use something like https://github.com/fullstorydev/grpcurl for changing the configuration of envoy instance on the fly? Or contour has some kind of configuration template?

[jpeach]

Assign to Nick to review.

[youngnick]

Thanks for logging this issue @wildermesser. I think there are a few questions in here, I'll do my best to answer as I can.

For your example, it looks like the more complex functionality you want to use allows specifying what the hash that goes into the cookie will look like - in this case, based on the user-agent and source IP. I can see how that would be useful, I hadn't taken the time to fully parse how this hash_policy works before - I thought you could only have one at a time. It does seem like it would be useful to be able to do things like this, and I can see we also have #2965 which describes a similar request. I have absolutely no idea what the way to configure this would look like though, this would almost certainly need a full design document, to figure out how to fit it into HTTPProxy. We're also constrained by the fact that HTTPProxy is a GA API, so we can't change the meaning of existing fields in incompatible ways.

You've also asked some other things about how to change Envoy's config - there are a couple of relevant points here:

• The initial bootstrap of Envoy is, in the example YAMLs, generated by contour bootstrap. You don't need to use this, you could generate your own bootstrap, and add any static config you want in there (this would apply to clusters and other objects, but not listener config).
• If you want to add extra config in to the listener (for example modifying the hash_policy), Contour owns that config, so Contour will probably need to make the change, unless you do what @bgagnon did and have a streaming xDS proxy that modifies things in flight. I probably don't recommend that approach now, it's probably easier at this point to get something that meets your need into Contour, then everyone can benefit.

I hope this answers some of your questions and look forward to hearing more from you.

[jpeach]

xref #2856

[wildermesser]

Thanks for answering my questions!

1. It is not clear from envoy documentation that you can use a list of parameters for hash_policy. Not only one. I did the same mistake when reading it for the first time. I agree that API shouldn't be braked. One possible solution was offered by
   Starblade42 here.
   We can add new strategy: strategy: RingHash which supports a list of options for hash_policy.
2. I thought that one can send a simple API query to envoy instance (yes, not HTTP but gRPC) to modify hash_policy. But the world turned out to be more complicated :). It seems like I should write my own xDS API server. So I prefer to fork and modify contour instead if I will not find another way
   Thank you again for your attention to my case!

[youngnick]

If we can, @wildermesser, I'd really like to understand your use case, so we can make it so you don't need to carry a fork (at least for too long). Are there specific things you're looking to achieve with the greater configuration flexibility?

[wildermesser]

No, nothing more than I specified on the topic message. I think that it is rational to implement all hash_policy options from evnoy because:

• declaration of all options is similar
• someone may require other options than I need. For example query_parameter

[youngnick]

I think that there's some work here to understand a number of requests, this one #2856, and #2965, which all interact with fine-tuning load balancing. We may need to add a new field that exposes the complexity, but this one will need a design document. The core maintainers don't have this one prioritised yet, but I think we'll try and get it on the roadmap.

[wildermesser]

Glad to hear it! I will be happy to review this design document if you consider it is useful.

[sunjayBhatia]

Relevant design: #3268

[sunjayBhatia]

See #3282 if you would like to review the changes going in to start supporting this

[sunjayBhatia]

we do support a list of request hash policies but so far have only implemented header based hashing

it is possible to add other types, including cookie based (envoy generated) and source ip, see https://github.com/projectcontour/contour/blob/main/design/loadbalancer-hash-policy-design.md#changes-to-loadbalancerpolicy for an example of how we could add a new field

we could deprecate the existing cookie load balancing functionality to migrate to supporting only the request hash strategy instead and move the cookie generation (and add passive cookie hashing) there so it can be used in conjunction with header hashing

[youngnick]

Yes, that sounds good. I think that we should investigate how we can add more of the hash policy flexibility from Envoy into the new hash policy list.

[sunjayBhatia]

heres a start of an API: #3993 any feedback appreciated

[sunjayBhatia]

we do support a list of different request attribute hash policies, you can hash a request on multiple attributes of a request.

we can open subsequent issues for new attributes to hash on that users want