Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EKS] Enable HPA with CloudWatch metrics and alarms #120

Open
joshuabaird opened this issue Jan 18, 2019 · 14 comments
Open

[EKS] Enable HPA with CloudWatch metrics and alarms #120

joshuabaird opened this issue Jan 18, 2019 · 14 comments
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue

Comments

@joshuabaird
Copy link

joshuabaird commented Jan 18, 2019

Tell us about your request
Many ECS customers make use of ECS service autoscaling based on CloudWatch metrics and alarms. This functionality is desired in EKS.

Community projects that add this support include https://github.com/chankh/k8s-cloudwatch-adapter

Which service(s) is this request for?
This could be EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We need to be able to use HPA to autoscale pods based on Cloudwatch metrics and alarms.

Are you currently working around this issue?
We use ECS.

@joshuabaird joshuabaird added the Proposed Community submitted issue label Jan 18, 2019
@mogren mogren added the EKS Amazon Elastic Kubernetes Service label Jan 22, 2019
@tabern tabern changed the title [EKS] [request]: Enable HPA with CloudWatch metrics and alarms [EKS] Enable HPA with CloudWatch metrics and alarms Jan 30, 2019
@tabern
Copy link
Contributor

tabern commented Mar 14, 2019

You can do this today in EKS by using custom metrics with the Kubernetes metrics server (part of the HPA implementation). These are typically metrics from within the Kubernetes cluster however and preform a similar function to how CW metrics and alarms work for ECS. I think what you are describing would be to consume CW metrics as external metrics into the metrics server and use these to trigger scaling.

Can you describe the use case that having this feature would enable?

@whereisaaron
Copy link

k8s/EKS has native support for autoscaling both application container replicas and cluster worker nodes based on cluster's built-in metrics collection. You don't need to integrate external systems like CloudWatch.

However, if you did still want to scale off CloudWatch, e.g. maybe you are triggering scaling inside EKS in response to SQS queue length, then you could trigger a Lambda that tells the EKS k8s API to scale up down your application (like kubectl scale from the CLI).

@whereisaaron
Copy link

whereisaaron commented Mar 14, 2019

That project mentioned @joshuabaird has disappeared. If you want to go the other direction, export internal cluster metrics to CloudWatch. then Istio has an adaptor for export traffic metrics and this issue #38 might be relevant.

@joshuabaird
Copy link
Author

Sorry - my point was that users migrating from ECS are most likely using CloudWatch metrics to autoscale their ECS services, so this same functionality is desired in EKS. It's an important feature to consider seeing that CloudWatch is a fundamental service in AWS, in my opinion.

Use-case:

We currently push custom metrics to CloudWatch for various things. These metrics are used to autoscale our ECS services. So, ideally, we need similar functionality in EKS otherwise we would need to re-write our entire metrics pipeline for this use case.

@whereisaaron
Copy link

Cool @joshuabaird, if you already have CloudWatch metrics then right now you can trigger:

  • Cluster scaling just by scaling the worker node ASG as normal, new nodes will automatically join the cluster, workload on deleted nodes gets automatically rescheduled
  • Service scaling using the CloudWatch -> Lambda -> EKS k8s API approach

The k8s built-in horizontal autoscaler also supports custom metrics, so someone or AWS could implement a CloudWatch metrics adaptor. Then the native k8s service scaling would be driven off CloudWatch metrics. There are existing projects for scaling based on Azure, Google, Datadog and Prometheus metrics. Adding a CloudWatch metrics adaptor seems like a good way to add what you want?

@joshuabaird
Copy link
Author

Yep, adding a Cloudwatch metrics adapter sounds like a solution!

@chankh
Copy link

chankh commented Mar 25, 2019

@whereisaaron @joshuabaird the project has been moved to https://github.com/awslabs/k8s-cloudwatch-adapter

@inductor
Copy link

inductor commented Aug 5, 2019

While I understand this is possible by using the described workaround, I believe HPA is a standard feature for Kubernetes and as for the metrics-server can be a default toolset for a managed Kubernetes cluster.

I'd really like this feature to come true without setting much!

@charlesparasa
Copy link

How can I push some of my Application Custom Metrics to Cloudwatch and use them in my HPA

@zsrinivas
Copy link

zsrinivas commented Apr 18, 2021

while the k8s cloudwatch adapter & eks container insights is great, it's still a major cluster level component if it stopped working for any reason then all autoscaling of all deployments stop, which is a significant risk.

While in ECS entire autoscaling is completely managed without any in-house management. this is one of the many reliability issues of managed aws managed k8s platform where it's not truly aws managed, there is a significant chunk of cluster-level components that need to be managed in-house. AWS managing such basic components will greatly increase the adoption of EKS from our team due to increased reliability.

@petty22
Copy link

petty22 commented Aug 25, 2021

Is there any updates on this? I have just spoke to AWS Support and they indicate that the k8s-cloudwartch-adapter is unsupported now. When can we expect a way to natively scale services based on cloudwatch metrics without have to do a run around for it?

@fbalicchia
Copy link

Hi k8s-cloudwartch-adapter seems to be archived but I can't find in favor of whom.
Could you point me please to an alternative ?

@sudip-moengage
Copy link

@fbalicchia use keda.sh

@eightnoteight
Copy link

cloudwatch metrics adapter doesn't seem like a good solution to use. major issue is that it uses GetMetricData api, which has rate limits, so if someone ran a script to fetch metrics for a usecase, suddenly you might see your autoscaling stopped working or worse started scaling down

the adapter either needs to maintain some state by itself or aws should expand the HPA to use an event driven approach like in ECS where alarm triggers an autoscaling action like KEDA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue
Projects
None yet
Development

No branches or pull requests