Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a resource: Discovery Engine Data Store #9860

Merged
merged 51 commits into from
Jan 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
55ef153
Create a configuration file product.yaml for product discoveryengine
jialei-chen Oct 24, 2023
c9928b9
Create resource DataStore configuration file template DataStore.yaml
jialei-chen Oct 24, 2023
d090272
Modify resource DataStore configuration file template to match the AP…
jialei-chen Oct 24, 2023
6e0aeff
Add each API resource field to configuration file properties attribute
jialei-chen Oct 25, 2023
58ae514
Add IAM support in Terraform to match DataStore API resource IAM support
jialei-chen Oct 25, 2023
45b5215
Polish field contents in product configuration file.
jialei-chen Nov 27, 2023
0741fa4
Modify DataStore.yaml for LRO handling, parameters matching and reada…
jialei-chen Nov 28, 2023
db93751
Add a DataStore create test.
jialei-chen Nov 28, 2023
3aa31ee
Modify product name from DiscoveryEngine to Discoveryengine so that g…
jialei-chen Nov 29, 2023
005dd1d
change to GA in doc (#9491)
edwardmedia Nov 27, 2023
451b7a1
Add value of to template into the generated tests to fix error 'fmt.…
jialei-chen Nov 29, 2023
5a3cdef
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Dec 18, 2023
c730698
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Dec 18, 2023
55c3d08
Remove Api::Product::ApiReference to match the change https://github.…
jialei-chen Dec 19, 2023
4c59446
Add var collection_id to unit test examples in resource config.
jialei-chen Dec 19, 2023
f400ae3
Fix argument missing error in unit tests.
jialei-chen Dec 19, 2023
e629831
Update resource create unit test case fields to contain display_name,…
jialei-chen Dec 19, 2023
3a73249
Fix errors in acceptance tests: now acc test fails at invalid parent …
jialei-chen Dec 20, 2023
c6ec24b
Set collection_id as default_collection.
jialei-chen Dec 20, 2023
e3b0ac8
Remove uncessary variables under unit test example.
jialei-chen Dec 20, 2023
75a7abc
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Dec 20, 2023
f605bac
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Dec 26, 2023
a7e591f
Fix acceptance tests issues by (1) removing iam_policy and (2) adding…
jialei-chen Dec 27, 2023
a6f6f17
Clean up configs: (1) update api version for tpg-beta to v1alpha; (2)…
jialei-chen Dec 27, 2023
fb879e9
Add examples and tests for all types of DataStore.
jialei-chen Dec 27, 2023
0ecd750
Remove redundant field project from create test examples.
jialei-chen Dec 27, 2023
7ac2058
Fix lint-yaml check failure.
jialei-chen Dec 27, 2023
e28ad41
Add field solution_types to acceptance tests to satisfy coverage requ…
jialei-chen Dec 27, 2023
14535d0
Clean up industry_vertical enum values.
jialei-chen Jan 2, 2024
5196ef6
Add a new query parameter createAdvancedSiteSearch to create_url.
jialei-chen Jan 2, 2024
d854bbb
Add a create test for advanced site search data store.
jialei-chen Jan 3, 2024
8158dd2
Add IAM roles/discoveryengine.admin.
jialei-chen Jan 8, 2024
d4172eb
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Jan 9, 2024
2236077
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Jan 9, 2024
7bf6eb0
Remove role config from .ci/infra/terraform/main.tf
jialei-chen Jan 9, 2024
765cdc9
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Jan 11, 2024
dae1f9e
Remove beta version in product.yaml because both ga and beta versions…
jialei-chen Jan 11, 2024
fb5c292
Rename product from Discoveryengine to DiscoveryEngine.
jialei-chen Jan 11, 2024
24dd57c
Remove unnecessary configs and improve description readability.
jialei-chen Jan 11, 2024
c0b90d7
Mark property solution_types as immutable.
jialei-chen Jan 11, 2024
f6b6309
Add update related settings and update test.
jialei-chen Jan 11, 2024
65587d0
Add delete_url for resource's Delete method.
jialei-chen Jan 11, 2024
c78f8d6
Remove all but one test example.
jialei-chen Jan 11, 2024
f573a23
Rename test example to make it more generic.
jialei-chen Jan 11, 2024
ae510b1
Modify first template in update test to omit all non-required fields.
jialei-chen Jan 11, 2024
2b29bde
Fix the update test: update display_name not data_store_id.
jialei-chen Jan 12, 2024
c26aa6b
Modify property description: remove brackets.
jialei-chen Jan 12, 2024
cc83a85
Update API version from v1alpha to v1.
jialei-chen Jan 22, 2024
149414a
Support regional APIs.
jialei-chen Jan 22, 2024
10f06cd
Merge branch 'GoogleCloudPlatform:main' into add-datastore-resource
jialei-chen Jan 23, 2024
850951f
Modify API reference link.
jialei-chen Jan 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 147 additions & 0 deletions mmv1/products/discoveryengine/DataStore.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Copyright 2023 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

--- !ruby/object:Api::Resource
name: 'DataStore'
description: |
Data store is a collection of websites and documents used to find answers for
end-user's questions in Discovery Engine (a.k.a. Vertex AI Search and
Conversation).
references: !ruby/object:Api::Resource::ReferenceLinks
guides:
'Create a search data store': 'https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-es'
api: 'https://cloud.google.com/generative-ai-app-builder/docs/reference/rest/v1/projects.locations.collections.dataStores'

base_url: 'projects/{{project}}/locations/{{location}}/collections/default_collection/dataStores'
self_link: 'projects/{{project}}/locations/{{location}}/collections/default_collection/dataStores/{{data_store_id}}'
create_url: 'projects/{{project}}/locations/{{location}}/collections/default_collection/dataStores?dataStoreId={{data_store_id}}&createAdvancedSiteSearch={{create_advanced_site_search}}'
update_verb: :PATCH
update_mask: true
delete_url: 'projects/{{project}}/locations/{{location}}/collections/default_collection/dataStores/{{data_store_id}}'
import_format:
[
'projects/{{project}}/locations/{{location}}/collections/default_collection/dataStores/{{data_store_id}}',
]

autogen_async: true
async: !ruby/object:Api::OpAsync
operation: !ruby/object:Api::OpAsync::Operation
path: "name"
base_url: "{{op_id}}"
wait_ms: 1000
timeouts: !ruby/object:Api::Timeouts
insert_minutes: 60
update_minutes: 60
delete_minutes: 60
result: !ruby/object:Api::OpAsync::Result
path: "response"
status: !ruby/object:Api::OpAsync::Status
path: "done"
complete: true
allowed:
- true
- false
error: !ruby/object:Api::OpAsync::Error
path: "error"
message: "message"

examples:
- !ruby/object:Provider::Terraform::Examples
name: "discoveryengine_datastore_basic"
primary_resource_id: 'basic'
primary_resource_name:
'fmt.Sprintf("tf_test_data_store%s", context["random_suffix"])'
vars:
data_store_id: "data-store-id"
parameters:
- !ruby/object:Api::Type::String
name: 'location'
required: true
immutable: true
url_param_only: true
description: |
The geographic location where the data store should reside. The value can
only be one of "global", "us" and "eu".
- !ruby/object:Api::Type::String
name: 'dataStoreId'
required: true
immutable: true
url_param_only: true
description: |
The unique id of the data store.
- !ruby/object:Api::Type::Boolean
name: 'createAdvancedSiteSearch'
default_value: false
url_param_only: true
description: |
If true, an advanced data store for site search will be created. If the
data store is not configured as site search (GENERIC vertical and
PUBLIC_WEBSITE contentConfig), this flag will be ignored.

properties:
- !ruby/object:Api::Type::String
name: 'name'
description: |
The unique full resource name of the data store. Values are of the format
`projects/{project}/locations/{location}/collections/{collection_id}/dataStores/{data_store_id}`.
This field must be a UTF-8 encoded string with a length limit of 1024
characters.
output: true
- !ruby/object:Api::Type::String
name: 'displayName'
description: |
The display name of the data store. This field must be a UTF-8 encoded
string with a length limit of 128 characters.
required: true
- !ruby/object:Api::Type::Enum
name: 'industryVertical'
description: |
The industry vertical that the data store registers.
values:
- :GENERIC
- :MEDIA
immutable: true
required: true
- !ruby/object:Api::Type::Array
name: 'solutionTypes'
description: |
The solutions that the data store enrolls.
item_type: !ruby/object:Api::Type::Enum
name: 'solutionType'
description: |
The type of solution.
values:
- :SOLUTION_TYPE_RECOMMENDATION
- :SOLUTION_TYPE_SEARCH
- :SOLUTION_TYPE_CHAT
immutable: true
- !ruby/object:Api::Type::String
name: 'defaultSchemaId'
description: |
The id of the default Schema associated with this data store.
output: true
- !ruby/object:Api::Type::Enum
name: 'contentConfig'
description: |
The content config of the data store.
values:
- :NO_CONTENT
- :CONTENT_REQUIRED
- :PUBLIC_WEBSITE
immutable: true
required: true
- !ruby/object:Api::Type::Time
name: "createTime"
description: |
Timestamp when the DataStore was created.
output: true
22 changes: 22 additions & 0 deletions mmv1/products/discoveryengine/product.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Copyright 2023 Google Inc.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

--- !ruby/object:Api::Product
name: DiscoveryEngine
display_name: Discovery Engine
versions:
- !ruby/object:Api::Product::Version
name: ga
base_url: https://{{location}}-discoveryengine.googleapis.com/v1/
scopes:
- https://www.googleapis.com/auth/cloud-platform
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
resource "google_discovery_engine_data_store" "basic" {
location = "global"
data_store_id = "<%= ctx[:vars]['data_store_id'] %>"
display_name = "tf-test-structured-datastore"
industry_vertical = "GENERIC"
content_config = "NO_CONTENT"
solution_types = ["SOLUTION_TYPE_SEARCH"]
create_advanced_site_search = false
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
package discoveryengine_test

import (
"github.com/hashicorp/terraform-plugin-sdk/v2/helper/resource"
"github.com/hashicorp/terraform-provider-google/google/acctest"
"testing"
)

func TestAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_update(t *testing.T) {
t.Parallel()

context := map[string]interface{}{
"random_suffix": acctest.RandString(t, 10),
}

acctest.VcrTest(t, resource.TestCase{
PreCheck: func() { acctest.AccTestPreCheck(t) },
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories(t),
CheckDestroy: testAccCheckDiscoveryEngineDataStoreDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_basic(context),
},
{
ResourceName: "google_discovery_engine_data_store.basic",
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"location", "data_store_id", "create_advanced_site_search"},
},
{
Config: testAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_update(context),
},
{
ResourceName: "google_discovery_engine_data_store.basic",
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"location", "data_store_id", "create_advanced_site_search"},
},
},
})
}

func testAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_basic(context map[string]interface{}) string {
return acctest.Nprintf(`
resource "google_discovery_engine_data_store" "basic" {
location = "global"
data_store_id = "tf-test-data-store-id%{random_suffix}"
display_name = "tf-test-structured-datastore"
industry_vertical = "GENERIC"
content_config = "NO_CONTENT"
}
`, context)
}

func testAccDiscoveryEngineDataStore_discoveryengineDatastoreBasicExample_update(context map[string]interface{}) string {
return acctest.Nprintf(`
resource "google_discovery_engine_data_store" "basic" {
location = "global"
data_store_id = "tf-test-data-store-id%{random_suffix}"
display_name = "updated-tf-test-structured-datastore"
industry_vertical = "GENERIC"
content_config = "NO_CONTENT"
solution_types = ["SOLUTION_TYPE_SEARCH"]
create_advanced_site_search = false
}
`, context)
}