Cache cleanup in S3 when using AWS EKS

[andrepiske]

I'm trying out Cirrus by using cirrus-ci.com through Github marketplace and using runners in my infra in AWS EKS.

When using the cache instruction, I see Cirrus created a S3 bucket and stores all cache data in there.

I'd like to know whether Cirrus does some management of the S3 bucket in terms of deleting old cache entries or something like that. I didn't see any lifecycle configuration in the bucket that would do this automatically.

[fkorotkov]

Hi Andre, Cirrus should've created "Delete old entries rule" with an expiration of 90 days for the bucket. Could you please check if it's there. Or maybe the associated credentials didn't have s3:PutLifecycleConfiguration so it wasn't created.

In either cases we don't do any additional cache management/eviction beside configuring lifecycle rules initially.

[andrepiske]

Hey!
Alright, thanks for the explanation and quick answer.

I double checked in the AWS S3 web UI and there's nothing under the bucket's "Management" tab ("There are no lifecycle rules for this bucket.", also no replication or inventory configurations). I also double checked the AWS user that Cirrus is using and it does have the s3:PutLifecycleConfiguration permission. In fact, it has all S3 permission listed under the blue box in the docs at https://cirrus-ci.org/guide/supported-computing-services/#configuring-aws-credentials

It could be that I didn't configure permissions properly for the user at the moment the bucket got created. So now I went ahead and deleted the bucket and triggered a new build (this one: https://cirrus-ci.com/build/6366311045922816) in the hopes it would re-create the S3 bucket. It did re-create the bucket but there isn't any lifecycle configuration for it either.

Seems like a bug? I'd be happy to aid in debugging this.

[fkorotkov]

I was able to find logs for your bucket creation and AWS API returned 403 Access Denied for some reason. Unfortunately AWS API doesn't allow to create a bucket and configure CORS/policies/lifecycle in one request like GCS can so Cirrus does it in sequential requests.

It seems you are not using OIDC so are you able to create a bucket and lifecycle rule for it using this creadentials?

Also this bucket will persist for the credentials you use, so all tasks using these credentials will end up with this bucket. You can manually add the lifecycle rule you want but please also add policy like:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::cirrus-ci-YOUR_BUCKET-us-east-1",
                "arn:aws:s3:::cirrus-ci-YOUR_BUCKET-us-east-1/*"
            ],
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": "false"
                }
            }
        }
    ]
}

As well as CORS:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "POST"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": []
    }
]

[andrepiske]

hey! You asked:

[...] so are you able to create a bucket and lifecycle rule for it using this creadentials?

Yes, and I think I found where the issue lies: it was missing the s3:PutBucketPolicy permission. I guess the code that configures the bucket does it in this order: first configure CORS, then Lifecycle, then bucket policy.

I added the s3:PutBucketPolicy permission to the user Cirrus talks to my aws account and went to do the re-create S3 bucket thing again. It then created the lifecycle configuration this time (I see a lifecycle rule named "Delete old entries rule" that deletes stuff after 90 days) and also the bucket policy. Previously, it had only the CORS rule configured. Thus, I guess it succeeded to create the CORS config but then failed to create the policy due to the missing permission and then didn't execute the create lifecycle config because the code flow stopped at the policy creation error.

If this is correct, I think the blue box in the Configuring AWS Credentials page should have the s3:PutBucketPolicy permission added to the list.

[fkorotkov]

Sorry I got distracted when I saw you message and just remembered that docs still not updated. You are 100% correct and documentation is now updated. Sorry for the inconvenience!