Support Custom SSM Alias

[meetreks]

Tell us about your request

Hi Team,

As per the code here https://github.com/aws/karpenter/blob/main/pkg/providers/amifamily/ami.go#L135 , SSM Alias for AMI seems to be handled but I understand speaking the the engineer that this is default SSM alias and hence custom SSM alias is not handled (yet). The documentation for this is available here -- https://docs.aws.amazon.com/autoscaling/ec2/userguide/using-systems-manager-parameters.html

Can I kindly request we can override with custom SSM alias to pick up the latest custom AMI to be used with Launch Template?

Regards,
Ramesh M

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

I would like Karpenter to be able to use Custom AMI from SSM as described here -- https://docs.aws.amazon.com/autoscaling/ec2/userguide/using-systems-manager-parameters.html

Are you currently working around this issue?

As this feature is not available, we are not able to work around this issue.

Additional Context

No response

Attachments

https://docs.aws.amazon.com/autoscaling/ec2/userguide/using-systems-manager-parameters.html

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

We support custom AMIs through the amiSelector in the AWSNodeTemplate. Could you tag the AMIs that you want to resolve through SSM alias?

[meetreks]

Hi @jonathan-innis Thanks for reverting back. Couple of things, 1. As SSM Aliases are currently supported - albeit for default AMI families, would it make sense to support custom AMI families as well?
2. The custom AMI selector under AWSNodeTemplate - can it support wildcards? The code seems to suggest its a string input but when we try with wildcards its nto able to return the right AMI. The same string query works fine when we try ec2 describe-images.

[jonathan-innis]

@meetreks This is something that we are going to take into consideration as part of #1327. We're still not sure if SSM aliases are the right way to go, though. Ideally, we would like to support a default tagging mechanism for the default AMIs such that amiSelector mechanism could work for both custom AMIs and default AMIs.

Which AMIs are you trying to resolve through SSM?

[meetreks]

@jonathan-innis Many thanks for reverting back. We create custom EKS AMI's to cater for different use cases - plain-vanila, with lustre, etc..These are the custom AMI(s) which we would like to be handled via SSM alias to keep it inline with Terraform and other method possible today.

[jonathan-innis]

keep it inline with Terraform and other method possible today

Is it possible to use tags in this case or change your terraform to also use tags. Could you tell me what the operational burden/overhead looks like for leveraging tags vs. leveraging SSM in your image deployment pipelines?

[kimxogus]

There's Custom amiFamily already, so I think we can use implement that

[meetreks]

There's Custom amiFamily already, so I think we can use that

Its dummy code, check it out.

[kimxogus]

I mean we can implement on that Custom amiFamily

[meetreks]

keep it inline with Terraform and other method possible today
Is it possible to use tags in this case or change your terraform to also use tags. Could you tell me what the operational burden/overhead looks like for leveraging tags vs. leveraging SSM in your image deployment pipelines?

This feature ensures that all our AMIs to be up to date, our enterprise policy means we have frequent AMI updates and Tenants/Clients are not always looking to update them as priority, this feature helps to automate this process using in conjunction with Drift functionality. It also reduces the need of any bespoke code to implement this on our side using for ex.. events, pipelines triggers and etc. And also, this feature is something that is supported on terraform side and hence its readily available with MNG with ASG , so we feel that Karpenter should also match this feature.

[jonathan-innis]

this feature is something that is supported on terraform side and hence its readily available with MNG with ASG

I'm mainly curious what would be the operational burden to add a specific tag to AMIs that Karpenter could use and what the operation burden of this might look like to use it on MNG and ASG.

I'm open to this feature add, I'm mainly just curious how much extra work the current state of the world is so that we can better understand the ask

[meetreks]

this feature is something that is supported on terraform side and hence its readily available with MNG with ASG

I'm mainly curious what would be the operational burden to add a specific tag to AMIs that Karpenter could use and what the operation burden of this might look like to use it on MNG and ASG.

I'm open to this feature add, I'm mainly just curious how much extra work the current state of the world is so that we can better understand the ask

The AMI vending is a separate process and is rigid in terms of adding custom tags for tennants and hence the request to keep it similar to the other options available today.

[meetreks]

Picked up this issue, will advise when completed.

[njtran]

@meetreks Hey how's it going? Feel free to reach out to us in the kubernetes slack if you have questions.

[meetreks]

@njtran just got to work on this, will ping you folks in slack for sure.

[ctaintor]

Just to add to this ticket – tags-on-AMI is not a usable solution if you use an AMI which is shared from another account. This is because AWS AMI sharing (and RAM in general) does not propagate the tags to the shared account.

Supporting ParameterStore natively is much preferred.

[benjimin]

Tags on AMIs seem an awkward way to indicate the AMI choice.

• You can't tag something that isn't owned by your account, but AMIs are commonly shared among accounts (like Amazon's EKS-optimised AMI is), so in general AMI tags aren't even an option.
• There have been recalled Amazon EKS-optimised AMIs in the last couple years; even organisations that use the stock AMI should have the capability to rollback to an earlier version than what Amazon is recommending at that moment (in case of some unanticipated incompatibility that impacts their specific deployments).
• It's cumbersome to promote a new AMI release, or to rollback to an earlier release, if you have to manipulate tags on multiple objects (both to mark the new choice and to unmark the previous choice).
• That design invites situations where either no AMIs or multiple AMIs are tagged, necessitating additional complexity to define how karpenter behaves in both of those situations. (This issue arises to some degree in the form of a race condition during every release or rollback, but is prone to becoming permanent if, for example, one tag change succeeds but the complementary tag change fails for any reason. To detect and handle this demands additional complexity in release/rollback automation, and it would obviously be a security problem if some karpenter deployments may be prone intermittently to silently assigning long-deprecated AMIs to new instances.)
• It is commonly intended that different clusters (prod and staging) deploy different releases from the same line of AMIs. So the choice should be stored someplace associated with the cluster itself, not associated to the AMI catalog like a global variable. (Conversely, it's messy if AMIs have multiple tags corresponding to every cluster that ever was, particularly if there is any dynamic provisioning and tear down of clusters..)

It makes more sense to simply store a reference to the chosen AMI (either in an SSM parameter, or a singular Kubernetes resource spec, or a tag attached to the EKS cluster rather than to the AMI). That way the updates are inherently atomic, and different clusters (e.g. prod vs staging) can implement different selections concerning the same range of AMIs.

[wesley-yep-canva]

Hey, just wanted to add to this ticket as well. We would also find it very useful as we have a multi-account strategy and sharing AMIs across accounts doesn't share the tags. Although we could build something to help sync the tags automatically, being able to use a ssm parameter would make it much easier to propagate AMI lifecycle changes.