EKS: backups fail when running on Bottlerocket nodes

[pedro-m-andrade-alb]

What steps did you take and what happened:

Cluster backups fail when the nodes use a Bottlerocket AMI, but not when they use Amazon Linux 2 AMI.

velero create backup backup-test --snapshot-move-data --exclude-namespaces actions-runner,actions-runner-controller,cert-manager,default,kube-node-lease,kube-public,kube-system

What did you expect to happen:
Backups finish without errors.

The following information will help us better understand what's going on:
We have some clusters running with Bottlerocket nodes and others with Amazon Linux 2 nodes. All backup tests in the Bottlerocket clusters failed, using the same command and configuration as the AL2 clusters, where the backups finished without errors.

Furthermore, we changed some AL2 clusters into Bottlerocket and the backups started to fail.

Dataupload error:
error to initialize data path: error to boost backup repository connection default-ffwk-kopia: error to connect backup repo: error to connect repo with storage: error to connect to repository: repository not initialized in the provided storage

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help
bundle-2024-10-29-10-23-43.tar.gz

Anything else you would like to add:

Environment:

• Velero version (use velero version):
  Client:
  Version: v1.14.1
  Git commit: 8afe3ce
  Server:
  Version: v1.14.1

• Velero features (use velero client config get features):
  features:

• Kubernetes version (use kubectl version):
  Client Version: v1.30.0
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.29.8-eks-a737599

• Cloud provider or hardware configuration:
  AWS EKS

• OS (e.g. from /etc/os-release):
  NAME=Bottlerocket
  ID=bottlerocket
  VERSION="1.25.0 (aws-k8s-1.29)"
  PRETTY_NAME="Bottlerocket OS 1.25.0 (aws-k8s-1.29)"
  VARIANT_ID=aws-k8s-1.29
  VERSION_ID=1.25.0
  BUILD_ID=388e1050
  HOME_URL="https://github.com/bottlerocket-os/bottlerocket"
  SUPPORT_URL="https://github.com/bottlerocket-os/bottlerocket/discussions"
  BUG_REPORT_URL="https://github.com/bottlerocket-os/bottlerocket/issues"

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

The backups failed because:
error to initialize data path: error to boost backup repository connection default-ffwk-kopia: error to connect backup repo: error to connect repo with storage: error to connect to repository: repository not initialized in the provided storage

Looks like something wrong happens to the object store and the repo data was not there.
You can delete the backuprepository CRs and retry the backup.

[pedro-m-andrade-alb]

We retried the backup with Bottlerocket nodes after deleting the backuprepository, and it failed with the error:
data path backup failed: Failed to run kopia backup: Failed to upload the kopia snapshot for si default@default:snapshot-data-upload-download/kopia/ffwk/grafana-pvc: permission denied

This test was done again in the same cluster but with AL2 nodes instead, and the backup finished successfully.

[kaovilai]

Probably related to #8249.

Please see fixes there and the new options you may need to apply.

[kaovilai]

Bottlerocket has an always-enabled, enforced, restrictive SELinux policy for the mutable filesystem that helps prevent containers from executing dangerous operations, even when running as root.

src

[sseago]

@kaovilai this is on 1.14 not 1.15, so I don't think the backupPVC issue is relevant. I think in this case, the --privileged-node-agent install flag may be all that's needed.

[pedro-m-andrade-alb]

Adding the --privileged-node-agent flag fixed the problem, and the backups now work with Bottlerocket nodes. Thank you all for the help. The issue can be closed.