Add details about the design of the redirector. #795
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
kind/kep
| @@ -109,6 +109,27 @@ Importantly, if a project needs to roll-back or remove an artifact, the same pro | |||
| occur, so that the promotion tool needs to be capable of deleting images and artifacts as | |||
| well as promoting them. | |||
|
|
|||
| ### HTTP Redirector Design | |||
| To facilitate world-wide distribution of artifacts from a single (virtual) location we will | |||
| run a replicated redirector service in the United States, Europe and Asia. Each of these redirector s | |||
There was a problem hiding this comment.
Nit: I'd probably add "initially" or "ideally" before "in the US, EU or Asia"
Also nit: space in redirectors
| To facilitate world-wide distribution of artifacts from a single (virtual) location we will | ||
| run a replicated redirector service in the United States, Europe and Asia. Each of these redirector s | ||
| services will be deployed in a Kubernetes cluster and they will be exposed via an External Service | ||
| behind a public IP address. |
There was a problem hiding this comment.
Do we need this detail? maybe just "Service of type LoadBalancer". And even that feels like an implementation detail.
| services will be deployed in a Kubernetes cluster and they will be exposed via an External Service | ||
| behind a public IP address. | ||
|
|
||
| We will use Geo DNS to route requests to `artifacts.k8s.io` to the correct redirector. We will need to extend or enhance the existing DNS synchronization tooling to handle creation of the GeoDNS records. |
There was a problem hiding this comment.
I think this means not using Google Cloud DNS, if we think geo is critical. It might be that if artifacts.k8s.io itself does a 302 redirect, we don't particularly care about geo on the first request. We could also encourage people to hit eu.artifacts.k8s.io or cn.artifacts.k8s.io if they need to force it for e.g. regulatory / firewall reasons.
But good call regardless - I know Route53 has Geo. Does Azure's DNS offer Geo routing? Or should we use something else entirely?
Personally I'd rather prefer to have records at multiple providers for resiliency, even if that means not having geo. i.e. ideally NS records that point to azure, to google, to aws etc. If one of those does want to implement geo-steering, that's great. It feels like a trade off between reliability and optimizing that first request though (and how many other providers we want to support)
There was a problem hiding this comment.
Yeah docs:
Is Google DNS visible in places like China? Even if it is, would the first hop to a Google hosted redirector work?
My main concern is that if the redirector IP isn't reachable then relying on 302 won't work.
There was a problem hiding this comment.
Yeah docs:
Is Google DNS visible in places like China? Even if it is, would the first hop to a Google hosted redirector work?
My main concern is that if the redirector IP isn't reachable then relying on 302 won't work.
There was a problem hiding this comment.
I think this is something we can figure out as we go. I was imagining that k8s.io would have NS records pointing to more than one of AWS, Azure & GCP. But as you point out, DNS resolution does not mean that the redirector HTTP/HTTPS service is reachable. And this isn't necessarily static either - the rules can change. We've captured the spirit anyway, and I don't think the point of the KEP process is to require the waterfall methodology :-)
| /kops | ||
| - Americas: americas.artifacts.k8s.io | ||
| - Asia: asia.artifacts.k8s.io | ||
| - default: americas.artificats.k8s.io |
There was a problem hiding this comment.
I like this. The "for now" also covers us for the future when I imagine we'll want to target by cloud also, to avoid cross-cloud bandwidth charges.
There was a problem hiding this comment.
Why do we need config for this? Why would an artifact have anything less than "full" mirroring?
| - default: americas.artificats.k8s.io | ||
| ``` | ||
|
|
||
| The redirector will us this data to proxy a requeast to the relevant mirror. The implementation of the mirrors themselves are details left to the service implementor and may be different depending on the artifacts being exposed (binaries s container images) |
There was a problem hiding this comment.
Nit: typo in request, and I think us should be use, s should be vs.
Also I think we should say "redirect a request", not "proxy a request". In my little PoC kubernetes/k8s.io#174 I was thinking we probably should proxy some trivial requests, I think the default should be a 302 redirect.
Reasons to proxy:
- we know the data is so small we might as well just serve it (somehow)
- the data is part of our chain of trust, and we don't want to trust a mirror. For example if we have a
.sha256checksum file, we probably want to just serve that if we can, rather than redirecting - particularly if we redirect to a non-https endpoint. I feel figuring out our chain of trust is a separate topic though!
|
A few nits / typos but lgtm. |
|
Comments addressed, please re-check. @justinsb let me know if/when you want to get started building :) |
|
@justinsb friendly ping on this one.... |
|
/approve (For whatever that's worth - I'm not crystal clear on who approves / lgtms this...) Also, first steps on building in kubernetes/k8s.io#174 - lots of structure around the world's most simple go program :-) |
|
Thanks @brendandburns! /lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: brendandburns, justaugustus, justinsb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
| services will be deployed in a Kubernetes cluster and they will be exposed via a public IP | ||
| address and a dns record indicating their location (e.g. `europe.artifacts.k8s.io`). | ||
|
|
||
| We will use Geo DNS to route requests to `artifacts.k8s.io` to the correct redirector. This is necessary to ensure that we always route to a server which is accessible no matter what region we are in. We will need to extend or enhance the existing DNS synchronization tooling to handle creation of the GeoDNS records. |
There was a problem hiding this comment.
GCP (our current DNS) doesn't currently support GeoDNS. If Geo is the only solution, we may have to overhaul that. Before calling for geo-dns as a solution, though, we should make sure we capture the requirements.
It sounds like you're aiming for:
- a single DNS name for end-users (rather than pushing geo onto them)
- support for mirrors other than Google Cloud (which has access problems in China)
- support for per-cloud mirrors
Are there more?
#1 can be addressed via anycast. If not for #2 and #3, that might be viable.
#2 might be mitigatable within the existing hosting. I have ideas on how to set that up, but it would need some testing, so we would need help from someone who actually suffers from the China blockages.
#3 is the real difficult one. It's not clear to me how to handle this - geodns is not quite the right model, since most clouds share most geographies, but it's still better to serve an aws user from within aws.
Lastly, I want to keep in mind that this all comes at some cost. Someone will have to foot the bills for this. Let's not downplay that hurdle. We should shoot for a model that is extensible to arbitrary mirrors, if possible.
Do any of the existing CDN products out there help us? Rather than build a CDN of our own...
There was a problem hiding this comment.
I don't actually think this is that hard, it's not really a CDN, we're not going to host the blobs (we'll use a real CDN for that) but we are going to host the redirector.)
When I said "GeoDNS" it means more than just "auto-geo"
Most clouds publish their IP ranges, e.g. AWS, Azure and GCP
So I think we can definitely determine which cloud a user is coming from and route accordingly.
There was a problem hiding this comment.
To further clarify:
GeoDNS gets us to the redirector. Redirector makes a decision based on cloud, and then Geo information.
There was a problem hiding this comment.
Yeah, I conflated a bit there. Sorry.
What we don't have here is requirements for the redirector itself. If this is to be in the path of every docker pull for k8s resources, it has to be DAMNED reliable.
One of the nice things about GCR/GCS (or s3 or ...) is that this becomes someone else's problem. Those services are understood to be pretty reliable, overall.
So we need a globally available redirector service with very high reliability and low latency. That redirector needs access to a federation of backend mirrors and rules to map incoming traffic to lists of viable mirrors.
We need some way to prove image veracity and non-tamperedness.
We need to treat this like a real production service. That means monitoring and on-calls and SLOs.
We probably need to health-check backend mirrors.
We need a notification and sync mechanism so backends can know when and what to mirror (e.g. rsync doesn't work against S3).
I don't know that we need GeoDNS to get to the redirector. We can stay within the existing infrastructure and use GCP's anycast capabilities and multi-cluster Ingress. it's my understanding that there's no problem with any of that from mainland China -- only things like googleapis.com. If we need to expand beyond that, we may need GeoDNS or something, yeah.
| /kops | ||
| - Americas: americas.artifacts.k8s.io | ||
| - Asia: asia.artifacts.k8s.io | ||
| - default: americas.artificats.k8s.io |
There was a problem hiding this comment.
Why do we need config for this? Why would an artifact have anything less than "full" mirroring?
@justinsb @thockin