AWS me-central-1 region is incompatible

[gaurav34065]

Thanos, Prometheus and Golang version used:

quay.io/thanos/thanos:v0.29.0
prometheus:v2.25.0

Object Storage Provider:
S3

What happened:
prometheus-thanos-compactor-0 pod is going in crashed state
What you expected to happen:
prometheus-thanos-compactor-0 pod should come up
How to reproduce it (as minimally and precisely as possible):
Use prometheus with thanos in region me-central-1 AWS
Full logs to relevant components:
level=info ts=2022-11-18T09:57:21.869909888Z caller=compact.go:619 msg="starting compact node"
level=info ts=2022-11-18T09:57:21.869935109Z caller=intrumentation.go:56 msg="changing probe status" status=ready
level=info ts=2022-11-18T09:57:21.870032667Z caller=intrumentation.go:75 msg="changing probe status" status=healthy
level=info ts=2022-11-18T09:57:21.870047073Z caller=http.go:73 service=http/server component=compact msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2022-11-18T09:57:21.87022067Z caller=tls_config.go:195 service=http/server component=compact msg="TLS is disabled." http2=false
level=info ts=2022-11-18T09:57:21.870274403Z caller=compact.go:1291 msg="start sync of metas"
level=debug ts=2022-11-18T09:57:21.870313714Z caller=fetcher.go:324 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=error ts=2022-11-18T09:57:23.713257529Z caller=compact.go:499 msg="retriable error" err="compaction: sync: BaseFetcher: iter bucket: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to."
level=debug ts=2022-11-18T09:57:23.713337594Z caller=fetcher.go:324 component=block.BaseFetcher msg="fetching meta data" concurrency=32
level=warn ts=2022-11-18T09:57:25.499375312Z caller=intrumentation.go:67 msg="changing probe status" status=not-ready reason="syncing metas: BaseFetcher: iter bucket: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to."
level=info ts=2022-11-18T09:57:25.499407542Z caller=http.go:84 service=http/server component=compact msg="internal server is shutting down" err="syncing metas: BaseFetcher: iter bucket: The me-central-1 location constraint is incompatible for the region specific endpoint this request was sent to."

Anything else we need to know:

[matej-g]

Looks like this is not related to Thanos but rather your S3 setup. This comment and thread in particular looks helpful: aws/aws-cli#2603 (comment).

[yeya24]

So the fix on minio has been released to v7.0.44, we just need to update our objstore repo and update the dependency here.

[yeya24]

Close this as we already updated thanos with the latest objstore.