IAM AssumedRole doesn't work in ECS

[bploetz]

I originally reported this in prometheus/cloudwatch_exporter#243, as I wasn't sure if this was an issue with the Prometheus cloudwatch exporter, or the underlying AWS Java SDK which it uses.

I'm trying to run the prom/cloudwatch-exporter:cloudwatch_exporter-0.7.0 Docker container on AWS ECS. The ECS Service is using an IAM Role Policy which includes the permissions:

cloudwatch:ListMetrics
cloudwatch:GetMetricStatistics
tag:GetResources

The following error occurs when starting the exporter:

[ec2-user@ip-X-X-X-X ~]$ sudo docker logs -f 5dbc385dade9
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.amazonaws.util.XpathUtils (file:/cloudwatch_exporter.jar) to method com.sun.org.apache.xpath.internal.XPathContext.getDTMManager()
WARNING: Please consider reporting this to the maintainers of com.amazonaws.util.XpathUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Dec 23, 2019 4:26:41 PM io.prometheus.cloudwatch.CloudWatchCollector collect
WARNING: CloudWatch scrape failed
com.amazonaws.services.cloudwatch.model.AmazonCloudWatchException: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/xxxxx-xxxxx-xxxxxx-xxxxxxx-xxxxx/xxxxxxxxxxxxxxx is not authorized to perform: cloudwatch:ListMetrics (Service: AmazonCloudWatch; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:2030)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1997)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1986)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executeListMetrics(AmazonCloudWatchClient.java:1323)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.listMetrics(AmazonCloudWatchClient.java:1295)
	at io.prometheus.cloudwatch.CloudWatchCollector.listDimensions(CloudWatchCollector.java:392)
	at io.prometheus.cloudwatch.CloudWatchCollector.getDimensions(CloudWatchCollector.java:346)
	at io.prometheus.cloudwatch.CloudWatchCollector.scrape(CloudWatchCollector.java:558)
	at io.prometheus.cloudwatch.CloudWatchCollector.collect(CloudWatchCollector.java:671)
	at io.prometheus.client.CollectorRegistry.collectorNames(CollectorRegistry.java:100)
	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:50)
	at io.prometheus.client.Collector.register(Collector.java:139)
	at io.prometheus.client.Collector.register(Collector.java:132)
	at io.prometheus.cloudwatch.WebServer.main(WebServer.java:27)

2019-12-23 16:26:41.727:INFO::main: Logging initialized @3281ms to org.eclipse.jetty.util.log.StdErrLog
2019-12-23 16:26:41.845:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826fa0b; jvm 11.0.5+10
2019-12-23 16:26:41.926:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@6a9d5dff{/,null,AVAILABLE}
2019-12-23 16:26:41.963:INFO:oejs.AbstractConnector:main: Started ServerConnector@79813403{HTTP/1.1,[http/1.1]}{0.0.0.0:9106}
2019-12-23 16:26:41.964:INFO:oejs.Server:main: Started @3519ms

It seems as if the assumed IAM Role attached to the container is not honored correctly.

I then tried adding the same IAM Role policy permissions noted above to the underlying EC2 host's IAM Role, and ran the exporter directly on the EC2 instance. This starts cleanly.

[ec2-user@ip-x-x-x-x ~]$ java -jar cloudwatch_exporter.jar 9106 /etc/config/cloudwatch_exporter/config.yml 
2019-12-23 16:28:49.379:INFO::main: Logging initialized @2385ms to org.eclipse.jetty.util.log.StdErrLog
2019-12-23 16:28:49.503:INFO:oejs.Server:main: jetty-9.4.z-SNAPSHOT; built: 2019-02-15T16:53:49.381Z; git: eb70b240169fcf1abbd86af36482d1c49826fa0b; jvm 1.8.0_222-b10
2019-12-23 16:28:49.565:INFO:oejsh.ContextHandler:main: Started o.e.j.s.ServletContextHandler@54eb2b70{/,null,AVAILABLE}
2019-12-23 16:28:49.581:INFO:oejs.AbstractConnector:main: Started Serv

I searched through the aws-sdk-java issues looking for issues pertaining to assumed roles within an ECS container and didn't find anything, but perhaps I missed it.

[bploetz]

To work around this, I created an IAM user with a policy containing the 3 permissions I noted above, and set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in the docker container for this user's key accordingly, and now the exporter starts up cleanly.

[max-rocket-internet]

It doesn't work on EKS either.

Using version 1.11.708.

The env vars on the pod are correct:

AWS_ROLE_ARN: arn:aws:iam::66666666666:role/xxxxxxxxxxx
AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

But it still tries to assume the role from instance instead.

[debora-ito]

@bploetz @max-rocket-internet I'm so sorry that this issue did not get any attention in a long time. Are you still experiencing the issue? If so please reply with a comment and I'll investigate.

Have you added aws-java-sdk-sts as a dependency?

[max-rocket-internet]

It works with aws-java-sdk-sts:1.11.724 so no longer an issue for us 🙂

[debora-ito]

@max-rocket-internet great!

[github-actions]

It looks like this issue hasn’t been active in longer than a week. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please add a comment to prevent automatic closure, or if the issue is already closed please feel free to reopen it.