Add data source for retrieving multiple GCS buckets from a project

[bestefreund]

Adds a new data source "data_google_storage_buckets", allowing to retrieve multiple GCS buckets from a project.

Fixes: hashicorp/terraform-provider-google#17845

Release Note Template for Downstream PRs (will be copied)

`google_storage_buckets`

[github-actions]

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@BBBmau, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 309 insertions(+))
google-beta provider: Diff ( 4 files changed, 309 insertions(+))

[modular-magician]

Tests analytics

Total tests: 92
Passed tests: 85
Skipped tests: 6
Affected tests: 1

Click here to see the affected service packages

• storage

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceGoogleStorageBuckets_basic

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataSourceGoogleStorageBuckets_basic[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[bestefreund]

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@BBBmau, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

@BBBmau
Please, take a look.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 311 insertions(+))
google-beta provider: Diff ( 4 files changed, 311 insertions(+))

[modular-magician]

Tests analytics

Total tests: 92
Passed tests: 85
Skipped tests: 6
Affected tests: 1

Click here to see the affected service packages

• storage

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceGoogleStorageBuckets_basic

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataSourceGoogleStorageBuckets_basic[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[bestefreund]

Tests failed during RECORDING mode: TestAccDataSourceGoogleStorageBuckets_basic[Error message] Debug log]

Please fix these to complete your PR. View the build log or the debug log for each test

@BBBmau
Thanks for starting.
Please, let me know what the issue is and if I can contribute anything to the solution.
Unfortunately, I have no access to the build logs of the VCR test.
A few moments ago I ran all tests in my local environment and every test, including TestAccDataSourceGoogleStorageBuckets_basic, ran smoothly without any issue.

TF_ACC=1 TF_SCHEMA_PANIC_ON_ERROR=1 go test ./google/services/storage -v -run=TestAccDataSourceGoogleStorageBuckets_ -timeout 240m -ldflags="-X=github.com/hashicorp/terraform-provider-google/version.ProviderVersion=acc"
=== RUN   TestAccDataSourceGoogleStorageBuckets_basic
=== PAUSE TestAccDataSourceGoogleStorageBuckets_basic
=== CONT  TestAccDataSourceGoogleStorageBuckets_basic
--- PASS: TestAccDataSourceGoogleStorageBuckets_basic (14.37s)
PASS
ok      github.com/hashicorp/terraform-provider-google/google/services/storage  14.416s

[BBBmau]

Took a look and it could be something to do with the ordering of buckets, this is the output from the error message:

 vcr_utils.go:152: Step 1/1 error: Check failed: Check 10/13 error: data.google_storage_buckets.all: Attribute 'buckets.0.name' expected "tf-bucket-test-1-0j9dezonpv", got "01utcv4zm2"

looking at the API responses both buckets are using the same name:

---[ REQUEST ]---------------------------------------
GET /storage/v1/b/tf-bucket-test-1-0j9dezonpv/o?alt=json&prettyPrint=false&versions=true HTTP/1.1
Host: storage.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.2.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google-beta/acc
X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
Accept-Encoding: gzip


-----------------------------------------------------
2024/04/17 17:35:21 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /storage/v1/b/tf-bucket-test-2-0j9dezonpv/o?alt=json&prettyPrint=false&versions=true HTTP/1.1
Host: storage.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.2.5 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google-beta/acc
X-Goog-Api-Client: gl-go/1.21.9 gdcl/0.171.0
Accept-Encoding: gzip

[bestefreund]

Thank you for taking a look and sharing the information.

I got a similar error when I was testing in my own project.

It originated in that the test project wasn't clean and contained already buckets, which was why the response contained unexpected results.

Cleaning up the test project from all buckets resolved the issue in my case.

Maybe, the test project from VCR-Test contains buckets that compromise the results.

Would you mind to clean up the test project and re-run the pipeline?
Or using another clean project?

[BBBmau]

looking more into this, this is one of those cases where we'd want to also create a project as part of the test so that the environment stays consistent. This makes it so that only the two buckets that are created here will only exist for this project that's also created as part of running the test.

This is done in other resources, an example of this is found here:

magic-modules/mmv1/third_party/terraform/services/storage/resource_storage_bucket_test.go.erb

Lines 2226 to 2231 in 5c9241e

	resource "google_project" "acceptance" {
	name = "tf-test-%{random_suffix}"
	project_id = "tf-test-%{random_suffix}"
	org_id = "%{organization}"
	billing_account = "%{billing_account}"
	}

[bestefreund]

This is a really good approach.
Thank you very much!

I'd like to implement it, but I'm restricted to a single project and cannot run the tests locally then.

[bestefreund]

I followed your suggestion and I've extended the test case with the deployment of it's own project to have a clean test environment.

Please, have a look on it.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 322 insertions(+))
google-beta provider: Diff ( 4 files changed, 322 insertions(+))

[modular-magician]

Tests analytics

Total tests: 92
Passed tests: 85
Skipped tests: 6
Affected tests: 1

Click here to see the affected service packages

• storage

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceGoogleStorageBuckets_basic

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataSourceGoogleStorageBuckets_basic[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 323 insertions(+))
google-beta provider: Diff ( 4 files changed, 323 insertions(+))

[modular-magician]

Tests analytics

Total tests: 93
Passed tests: 86
Skipped tests: 6
Affected tests: 1

Click here to see the affected service packages

• storage

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceGoogleStorageBuckets_basic

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataSourceGoogleStorageBuckets_basic[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[bestefreund]

@BBBmau
Please, restart the pipeline and have a look again.

I've fixed a typo and the acceptance test with the included project succeeded with my development environment.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 4 files changed, 323 insertions(+))
google-beta provider: Diff ( 4 files changed, 323 insertions(+))

[modular-magician]

Tests analytics

Total tests: 93
Passed tests: 86
Skipped tests: 6
Affected tests: 1

Click here to see the affected service packages

• storage

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceGoogleStorageBuckets_basic

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccDataSourceGoogleStorageBuckets_basic[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

---

$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

[BBBmau]

All looks great, Thanks for the contribution!

[bestefreund]

Thanks for your patience and the review!