GCS Soft Delete Announcement / Terraform interaction

[joe-a-t]

I just saw https://cloud.google.com/resources/storage/soft-delete-announce?hl=en and this raised a lot of questions about how Terraform will interact with deleting both objects and buckets once soft delete is enabled by default by GCP. Can provider maintainers please weigh in on how you plan to interact with GCS once this goes into effect?

1. Will Terraform succeed in deleting an object/bucket if it is in soft delete state or will the object/bucket continuing to exist in that soft delete state instead of having been entirely deleted cause Terraform to fail?
2. How will Terraform respond when it goes to delete a bucket if all objects in the bucket are in the soft delete state?
3. Will force_destroy continue to work? Looks like no based on my understanding.
4. Will the Terraform Google Provider set any default retention period settings that contradict the new default retention settings Google will be setting as part of this announcement?
5. If the Terraform Google Provider does not set a default retention period, will the provider be modified to gracefully accept the new default value Google is setting?
6. etc

Feel free to modify this at will or replace with your own issue/post covering these issues, just trying to throw these out a starting point.

b/326044671

[shuyama1]

Thanks @joe-a-t for taking time to submit your questions and for opening this ticket. Seeing as this feature has not been launched officially, I will answer the questions I can with the knowledge I have now, and I will also be forwarding this ticket to the service team so they can provide their input.

1. Will Terraform succeed in deleting an object/bucket if it is in soft delete state or will the object/bucket continuing to exist in that soft delete state instead of having been entirely deleted cause Terraform to fail?

With the soft-delete feature being launched for GCS, I do not think we anticipate any significant changes to how Terraform handles deletion of these resources. Terraform should still be able to delete an object or bucket with a soft-delete policy enabled (which is the default setting). This means that running terraform destroy will not result in an error with soft delete enabled and the resource will simply enter the soft-delete state. One possible change is that a new resource may not be created if it has the same name as an existing resource that is still in the soft delete retention period.

2. How will Terraform respond when it goes to delete a bucket if all objects in the bucket are in the soft delete state?

I think it depends heavily on how the API handles this scenario. Terraform will generate DELETE call to the API and pass back whatever response the API returns.

3. Will force_destroy continue to work? Looks like no based on my understanding.

This also depends heavily on the implementation details of the feature at the backend. Based on the information I have now, it seems that even with force_destroy, the resource will still be soft deleted unless the soft deletion is explicitly disabled.

4. Will the Terraform Google Provider set any default retention period settings that contradict the new default retention settings Google will be setting as part of this announcement?

No, we do not plan to set a client-side default for the soft delete retention period.

5. If the Terraform Google Provider does not set a default retention period, will the provider be modified to gracefully accept the new default value Google is setting?

Once the feature has been rolled out, Terraform Google Provider will honor the service-side default for the retention period. Soft-delete period related fields will need to be implemented in the providers to support customizing soft delete settings via Terraform after the API supports it.

[joe-a-t]

I think there is some confusion here? As far as I understand it, all of the relevant service side APIs exist today and you can test the exact performance of the proposed change today by explicitly setting a retention policy of 7 days on a bucket via a retention_policy. The contents of the announcement is not as much a new feature as a change of the default value.

[shuyama1]

My understanding is that the Soft Delete feature is different from the current retention_policy.

The current one is to ensure that the objects cannot be deleted at all until they reach the retention period specified. Before that, attempts to delete the objects will result in a failed deletion with the 403 - retentionPolicyNotMet error, according to Bucket retention policies.

The soft delete feature is, if enabled, after the object is deleted successfully, it goes into the soft-deleted state ("Soft-deleted objects are special non-readable objects that are hidden from object listings unless a specific option is specified"), with the capability to be restored until the retention ends, according to the soft delete announcement. The soft delete retention period means how long the object will be retained in the soft-deleted state before it's permanently deleted.

[joe-a-t]

Yeah, I think the normal retention policy is only for the "LIVE" version of an object since it's not compatible with object versioning. This new soft delete thing seems like it is basically a retention policy on "ARCHIVED" versions? It's definitely weird that it is disabled via reusing the existing retention policy arguments if it behaves differently than retention policy currently does.

So I guess it is super unclear if these soft deleted objects will behave in the same way as "ARCHIVED" versions of objects when it comes to using Terraform to delete a bucket with force_destroy = false, which currently fails with

Error: Error trying to delete bucket test containing objects without `force_destroy` set to true

if you try to delete a bucket that does not have any "LIVE" objects but does have some "ARCHIVED" objects. And there are a couple of possible ways I could see this being handled like:

1. Being unable to delete the bucket regardless of method until all objects have been deleted and the soft delete period on all objects has expired (most similar to current behavior if trying to delete a bucket that contains objects where the retention period is not met but appears to not be the case due to the last part of How restores work?)
2. Being able to delete the bucket via Terraform but only if you use force_destroy = true when there are not any "LIVE" objects but are still objects in the soft delete period.
3. Being able to delete the bucket via Terraform with force_destroy = false when there are still soft deleted objects in the bucket but no "LIVE" objects.

Of these options, #3 would probably be my personal preference and the least impactful to current Provider behavior although depending on how exactly it was implemented on the Google API side, the Provider may need to be modified so that it responds with a "yes I'm sure" upon receiving a warning that the bucket still contains soft deleted objects but no "LIVE" objects.

And it sounds like for any information on how this would function, we are stuck waiting on the Google service team to provide feedback on how their changes will affect the relevant APIs and if those API changes result in any kinds of changes on the Terraform Provider side.

[shuyama1]

Changed the label to bug, as I just realize that questions are not properly forwarded internally.

Note to the service team: This particular ticket is not a bug but rather a question. We're classifying it as such to ensure proper forwarding, as our internal system does not have a corresponding ticket type. We’d like to invite the service team to provide insight on the upcoming soft deletion feature in GCS and its impact on current Terraform users, specifically whether it will affect the current Terraform deletion process. In Terraform Google Provider, bucket deletion method offers an option force_destroy to whether to force delete a bucket by deleting all contained objects first, and then makes direct API calls to delete the bucket.

To better understand the potential impact for bucket deletion in Terraform, the more specific questions are:

1. Does LIST also show soft-deleted objects? It seems no based on the announcement
2. Does DELETE succeed for deleting a bucket containing only soft-deleted objects? If no, this will break the current Terraform deletion.

Additionally, it seems that soft deletion will be enabled by default for both existing and new buckets/objects. Please be aware that this means users won't be able to turn off soft-deletion via Terraform until the feature is implemented into the terraform providers as well.
Many thanks!

[kautikdk]

Hi @joe-a-t , @shuyama1. In response to above discussion thread, I like to highlight some technical aspects of upcoming soft delete feature.

1. Soft delete is not introducing any breaking changes to Cloud Storage.
2. Buckets that contain soft-deleted objects can still be deleted, just as before. It is different than retention policy which can prevent bucket deletion in some cases.
3. The list API will not return soft-deleted objects unless the new softDeleted query parameter is specified.
4. Any settings you provide in a Terraform will override service defaults.
5. There will be client side default of soft delete retention period which will take effect if the soft delete block is specified otherwise server side value will be kept.

Let me know if it answer the questions or something is missing!

[joe-a-t]

@kautikdk Thank you. Can you please get this information clarified on https://cloud.google.com/resources/storage/soft-delete-announce?hl=en?

In particular,

was quite confusing in how it appears to now be talking about a separate soft-delete retention period, not the existing bucket retention period. Given what appears to be huge differences in functionality, I'm quite worried about having identical vars that have very different purposes (and cost implications) depending on what specific block they are in (although granted it is unclear if this would be implemented as a block or a straight up argument like soft_delete_retention_period in the provider interface - I would guess historically a block given that most arguments on the provider are implemented as blocks even if they only have one argument inside the block like versioning, encryption, and custom_placement_config). Might be better to change the name (maybe switch retention period to duration) or some other clarification.

Additionally, it would be helpful if the Terraform link at the bottom of the announcement linked to an open issue or PR on the Terraform provider that had further details about how it will be implemented or links to the docs/implementation on the google-beta provider instead of the generic Google page for "What is Terraform"

[kautikdk]

@joe-a-t Sure, These information will be clarified on documentation. More technical details will be available at launch time which is in near future.

In the above documentation it refers to soft delete retention period. Generally Soft delete terraform support will be implemented as a new block as new terraform fields should mirror the API’s structure to ease predictability and maintenance. Inside that block there will be one field which can be used to specify soft delete duration in seconds. I will redirect your feedback regarding naming to appropriate team.

Regarding Terraform link, It is linked for tool reference rather than feature support reference. I am not sure about providing any link to open issue/PR/docs on terraform provider because providing terraform support for feature might have different priority considering CLI and Cloud Console support.

Thank you for your valuable feedback and views!

[NissesSenap]

So in the announcement, March was mentioned as the launch.
I can see soft delete in the UI, and it links to a document https://cloud.google.com/storage/docs/soft-delete, that shows 404.

@kautikdk any status update on this feature, and when this feature is coming to terraform would be great.
I have multiple places where this will need to be disabled, and currently GCP isn't giving us much time to do it.

[kautikdk]

Hi @NissesSenap, Terraform support will be provided after full GA of the feature but It will be before promotional time ends mentioned in the announcement which is May 31, 2024.
Regarding the link, I will redirect it to the respective team working on this.

[NissesSenap]

To be honest, @kautikdk , that don't sit too good with me and I will reach out to our sales contacts to discuss this.

GCP are doing a big change in its API and you are giving us very limited timeframe to adjust to that.
If you at least can add this to the beta provider we will at least have more time to do some tests around this.
Since I see it in the UI, I assume that the API is already there.

In short, the quicker this can be implemented so we as your customers can start to try this the better. And we would be very grateful if this can be solved as quickly as possible.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.