Bucket creation in recently opened AWS regions fails

[anothertobi]

Description

For recently opened AWS regions, bucket creations through restic fail with a 400 Bad Request during the BucketExists check.

Additional Context

minio-go defaults to the s3.dualstack.us-east-1.amazonaws.com endpoint when an unknown region is provided. The authorization header however still contains the provided, unknown, region. This results in a 400 Bad Request when calling the S3 API.

New regional S3 endpoints were added to minio-go in minio/minio-go#1726 and released with version v7.0.44 (makes eu-central-2 a "known" region). This version was included in restic in restic/restic@f4d3ed7 and released with version v0.15.0.

K8up currently uses github.com/restic/restic v0.14.0 and already has a PR pending to upgrade restic to v0.16.0 #796 (fixes the issue).

Logs

68d0d8e880ce:/$ k8up restic
2023-10-02T13:40:04Z	INFO	k8up	Starting k8up…	{"version": "2.7.1", "date": "2023-05-12T12:32:54Z", "commit": "81d9c2bd7970208bcb0f84f7141d7716177b92cb", "go_os": "linux", "go_arch": "arm64", "go_version": "go1.19.9", "uid": 65532, "gid": 0}
2023-10-02T13:40:04Z	INFO	k8up.restic	initializing
2023-10-02T13:40:04Z	INFO	k8up.restic	setting up a signal handler
2023-10-02T13:40:04Z	INFO	k8up.restic.restic	using the following restic options	{"options": [""]}
2023-10-02T13:40:04Z	INFO	k8up.restic.restic.RepoInit.command	restic command	{"path": "/usr/local/bin/restic", "args": ["init", "--option", ""]}
2023-10-02T13:40:04Z	INFO	k8up.restic.restic.RepoInit.command	Defining RESTIC_PROGRESS_FPS	{"frequency": 0.016666666666666666}
2023-10-02T13:40:06Z	INFO	k8up.restic.restic.RepoInit.restic.stderr	Fatal: create repository at s3:https://s3.eu-central-2.amazonaws.com/mybucket failed: client.BucketExists: 400 Bad Request
2023-10-02T13:40:06Z	INFO	k8up.restic.restic.RepoInit.restic.stderr	
2023/10/02 13:40:06 unable to start k8up: failed to initialise the restic repository: cmd.Wait() err: 1

Expected Behavior

The packaged restic binary in the K8up container image can create an S3 bucket in the eu-central-2 AWS region.

Steps To Reproduce

docker run -it --entrypoint /bin/bash ghcr.io/k8up-io/k8up:v2.7.1

# note: new region eu-central-2
export RESTIC_REPOSITORY=s3:https://s3.eu-central-2.amazonaws.com/mybucket
export RESTIC_PASSWORD=foobar
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=

k8up restic

Version of K8up

v2.7.1

Version of Kubernetes

1.27

Distribution of Kubernetes

EKS

[anothertobi]

@zugao could you maybe have a look at this one (#796 is the fix) and release a new K8up version?

[zugao]

@anothertobi just released v2.7.2. Can you let me know if the problem is fixed?

[anothertobi]

With K8up v2.7.2 the S3 bucket creation and backups in the AWS region eu-central-2 work as expected. Thanks for the new release 🎉