LD Relay Proxy failing to connect to DynamoDB

[yj10545]

I have setup LD Relay proxy with DynamoDB persistent storage in AWS with ESC and Application load balancer. I see following error msg when the Relay initialize -

2021/02/02 23:43:37.182131 ERROR: [env: ...0b42] Data store returned error: failed to get existing items prior to Init: NoCredentialProviders: no valid providers in chain. Deprecated.
2021/02/02 23:43:37.182160 WARN: [env: ...0b42] Detected persistent store unavailability; updates will be cached until it recovers
2021/02/02 23:43:37.182192 WARN: [env: ...0b42] Unexpected data store error when trying to store an update received from the data source: failed to get existing items prior to Init: NoCredentialProviders: no valid providers in chain. Deprecated.

I have given ECS permissions to read from and write to DynamoDB. Am I missing some settings? Pls. help.

[eli-darkly]

It does seem like a setting is missing, but I don't think it is necessarily a Relay Proxy setting. The "NoCredentialProviders: no valid providers in chain" error is coming from the AWS SDK, indicating that it wasn't able to find AWS credentials for accessing DynamoDB.

Relay Proxy doesn't itself do any configuration of AWS credentials; it relies on the "default credential provider chain" behavior of the AWS SDK as described here. If it is getting this error, then (as I understand it) any application trying to access DynamoDB via the AWS SDK from that same environment would also get that error. So whatever you did to make it so your ECS containers could access DynamoDB, it sounds like it didn't have the intended effect.

It is still possible that this is somehow a Relay Proxy problem rather than an AWS configuration problem, but as a maintainer of the Relay Proxy code I can't think of how that could be. It's possible that the LaunchDarkly support team has worked with other customers who use ECS and may be able to help you better, if you go through support.launchdarkly.com. AWS support might also be worth trying.

[eli-darkly]

Actually, is it possible that this is the same thing as #127? I'm not sure whether IRSA is relevant to this type of deployment, but if so, that will be addressed by an update we're about to do.

[yj10545]

I was able to resolve this issue after going through AWS documentation. The mistake I made was that I assigned the newly created DynamoDB policy to 'execution_role_arn' of the ECS task. I needed to assign that DynamoDB policy to 'task_role_arn' of the ECS task. After making this change LD populated my DynamoDB table. Thank you eli-darkly for your quick response. Its awesome that you responded so quickly, I really appreciate it.

[eli-darkly]

Great, glad to hear it. Sorry I didn't have more specific advice as we haven't used ECS in this way before.