S3 object store always returns IMDS warnings

[ion-elgreco]

When using the S3 Object store, it will still try to use the IMDS provider by default, only when setting an environment variable AWS_EC2_METADATA_DISABLED, it's not throwing the warnings anymore.

[2024-04-24T07:55:28Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: No route to host (os error 113): tcp connect error: No route to host (os error 113): No route to host (os error 113) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 113, kind: HostUnreachable, message: "No route to host" })), connection: Unknown } }) }))
[2024-04-24T07:55:29Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: timeout: error trying to connect: HTTP connect timeout occurred after 1s: HTTP connect timeout occurred after 1s: timed out (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 1s }), connection: Unknown } }) }))

@mightyshazam created an issue, so we can track this

[rtyler]

@ion-elgreco I assume this is manifesting when using the S3 backend outside of the AWS environment?

[ion-elgreco]

@ion-elgreco I assume this is manifesting when using the S3 backend outside of the AWS environment?

Yup! With custom S3 endpoints

[ion-elgreco]

@rtyler actually, also with just aws S3. Forgot that I was seeing that when @mightyshazam and I were chatting about it

[mightyshazam]

We were debugging it over Slack. We were both running the same code via command line and notebooks, but I wasn't seeing the error while Ion was. Ion's coworkers were also seeing it. We even accounted for potentially different environment variables, but that had no effect. I need more sample data to reproduce it.